CN116303500A - Data consistency verification method, device, equipment and storage medium - Google Patents

Abstract

The invention provides a data consistency verification method, a device, equipment and a storage medium, wherein the method comprises the following steps: dividing the verification period according to the number of the sampling time periods to obtain the sampling time periods, and executing a core inquiry script of the execution times of the core script in the sampling time periods; according to a hierarchical sampling algorithm, sampling data of a source database and a target database which are queried by a core query script at N time points in each sampling time period, and performing data volume consistency verification to obtain a data volume consistency verification result; and if the data volume consistency check result is that the data volume is consistent, carrying out data content consistency check on the data of the source database and the target database inquired by the core inquiry script to obtain a data content consistency check result. In the method, sampling verification is carried out on each time period by selecting a specific number of time points through layered sampling, so that the pressure of the database is reduced, the data problem is found in time through periodic verification, the verification effectiveness is improved, and the pressure of the database is reduced.

Description

Data consistency check method, device, equipment and storage medium

technical field

The invention relates to the technical field of databases, in particular to a data consistency checking method, device, equipment and storage medium.

Background technique

As businesses grow, enterprise systems become increasingly complex. The complex distributed system architecture is prone to abnormalities such as remote call failure and message sending failure, which may lead to data inconsistency between systems. In addition, as time goes by, the number of users of the system continues to grow, and the amount of data in the system is getting larger and larger. In order to ensure the performance of core functions while retaining historical data, the storage layer has gradually evolved into two architectures, hot and cold. For hot data, cold storage stores historical data, which also poses a challenge to data consistency between cold and hot storage.

The commonly used redundant data consistency check scheme in the prior art generally adopts the online mode, that is, a field is specially set in the database table to save the checksum of the record, and the checksum is updated at the same time when the record is updated. Checksum fields to check for consistency. In addition, based on the characteristics of the business, the offline mode may also be adopted, mainly by verifying the quantity of the original data to ensure the consistency of the quantity. The redundant data consistency verification scheme for the offline mode is decoupled from the business, but The effectiveness is low, and it will bring additional pressure to the database in the storage layer.

Contents of the invention

The main purpose of the present invention is to solve the technical problem that the existing redundant data consistency check scheme in offline mode has low effectiveness and will bring additional pressure to the database of the storage layer.

The first aspect of the present invention provides a method for verifying data consistency, the method comprising:

Obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles, and sampling time periods, and the core query scripts are used to query the source database and target database data;

Dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the execution times of the core script in the sampling time period;

According to the preset hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain consistent data volume A sex check result, wherein said N is a natural number not less than 1;

If the data volume consistency verification result is that the data volume is consistent, then perform data content consistency verification according to the data of the source database and the target database queried by the core query script, and obtain a data content consistency verification result;

A data consistency verification result is obtained according to the data content consistency verification result.

The second aspect of the present invention provides a data consistency verification device, including:

An acquisition module, configured to acquire preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include core script execution times, verification cycles and the number of sampling time periods, and the core query scripts Data used to query the source and target databases;

A query module, configured to divide the verification cycle according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the execution times of the core script in the sampling time period;

The data volume verification module is used to take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script according to a preset layered sampling algorithm, and perform data volume consistency Consistency verification, to obtain the consistency verification result of the data volume, wherein, the N is a natural number not less than 1;

A data content verification module, configured to perform a data content consistency verification according to the data of the source database and the target database queried by the core query script when the result of the data volume consistency verification is that the data volume is consistent, to obtain Data content consistency check result;

A verification result generating module, configured to obtain a data consistency verification result according to the data content consistency verification result.

The third aspect of the present invention provides a device for checking data consistency, including: a memory and at least one processor, instructions are stored in the memory, and the memory and the at least one processor are interconnected through a line; the At least one processor invokes the instructions in the memory, so that the data consistency checking device executes the steps of the above data consistency checking method.

A fourth aspect of the present invention provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on a computer, it causes the computer to execute the steps of the above data consistency checking method.

The above data consistency verification method, device, equipment and storage medium, by dividing the verification period according to the number of sampling time periods, obtain the sampling time period, and execute the core query script of the execution times of the core script in the sampling time period; The layer sampling algorithm takes N time points in each sampling period to sample the data of the source database and the target database queried by the core query script, and performs data volume consistency verification to obtain the data volume consistency verification result; if the data If the volume consistency verification result shows that the data volume is consistent, then the data content consistency verification is performed on the data in the source database and the target database queried by the core query script, and the data content consistency verification result is obtained. In this method, a specific number of time points are selected for sampling and verification in each time period through stratified sampling, thereby reducing the pressure on the database, and timely discovering data problems through periodic verification, improving the effectiveness of verification and reducing the pressure on the database.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

In order to make the above-mentioned objects, features and advantages of the present invention more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

Fig. 1 is the schematic diagram of the first embodiment of the data consistency checking method in the embodiment of the present invention;

Fig. 2 is a schematic diagram of an embodiment of a data consistency checking device in an embodiment of the present invention;

FIG. 3 is a schematic diagram of another embodiment of the data consistency checking device in the embodiment of the present invention;

Fig. 4 is a schematic diagram of an embodiment of a data consistency checking device in an embodiment of the present invention.

Detailed ways

The embodiment of the present invention provides a data consistency verification method, device, device, and storage medium. By dividing the verification cycle according to the number of sampling time segments, the sampling time segment is obtained, and the execution times of the core script are executed in the sampling time segment. Core query script; according to the hierarchical sampling algorithm, take N time points in each sampling period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain data volume consistency Verify the result; if the result of the data volume consistency verification is that the data volume is consistent, then the data content consistency verification is performed on the data of the source database and the target database queried by the core query script, and the data content consistency verification result is obtained. In this method, a specific number of time points are selected for sampling and verification in each time period through stratified sampling, thereby reducing the pressure on the database, and timely discovering data problems through periodic verification, improving the effectiveness of verification and reducing the pressure on the database.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the term "comprising" or "having" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to those explicitly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

For ease of understanding, the following describes the specific process of the embodiment of the present invention, please refer to Figure 1, the first embodiment of the data consistency verification method in the embodiment of the present invention includes:

101. Obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles, and sampling time periods, and the core query scripts are used to query the source database and target database data;

In this embodiment, the staff configures the data consistency verification task in advance. The data consistency verification task includes the data consistency verification parameters and the core query script set by the staff. When the server or system or data consistency verification When the device executes the process of this solution, it will obtain data consistency verification parameters and core query scripts from the data consistency verification task configured by the staff. Among them, the data consistency verification parameters include source database and target database addresses, core script execution Number of times, verification period, number of sampling interval time periods and retries, etc. In addition, other parameters about data consistency verification can also be included, which is not limited in the present invention. The core query script is used to query the source database and the target database The data,

In practical applications, the source database and the target database are two databases that need to maintain data consistency. For example, in order to ensure the performance of core functions while retaining historical data, the storage layer gradually evolves into two architectures, hot and cold, and the hot database stores core popular data. , the cold storage stores historical data, the hot storage can be used as the source database, and the cold storage can be used as the target database.

102. Divide the verification cycle according to the number of sampling time periods to obtain the sampling time period, and query the core query script according to the execution times of the core script in the sampling time period;

In this embodiment, the verification period is the period of time during which data consistency verification is required. For example, the verification period is 1 hour, which is 1 hour after the server or system or data consistency verification device executes the process of this solution. It is necessary to verify the data consistency between the source database and the target database multiple times, and divide the verification period according to the number of sampling time periods. For example, if the sampling time period is 5 and the verification period is 1 hour, then the verification period Every 12 minutes is used as a sampling time period. In each sampling time period, the core query script is executed to query the data of the source database and the target database, and the number of executions is the core script execution of the pre-set data consistency inspection parameters. The number of times, the data consistency problem between the source database and the target database is discovered in time through periodic verification.

103. According to the preset hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and check the consistency of the data volume to obtain a consistent data volume gender check result;

In this embodiment, in the process of data consistency verification, the data volume consistency verification is first performed, because the data content consistency verification needs to be encoded, and the data content has been compared one by one. The resources used are relatively large, so the data volume consistency check is performed first. When the data volume is inconsistent, it can be directly determined that the data in the source database and the target database are inconsistent. In the process of data volume consistency check, in order to further To reduce the pressure on the database, it does not directly compare the amount of data counted in each sampling time period, but takes a preset number of time points from the sampling time period through hierarchical sampling to obtain the source database queried by the core query script Compare with the data of the time point in the data of the target database. When the verification result is that the data volumes of these time points sampled by the two are consistent, the next step of verification is performed. If they are not consistent, the delayed retry try.

In this embodiment, according to the preset layered sampling algorithm, N time points are taken in each sampling time period to sample the data of the source database and the target database queried by the core query script, and the data volume Consistency check, after obtaining the data volume consistency check result, also includes: according to the core query script of execution times of core script execution times in each sampling time period, inquire about the query data volume of the data of the source database and the target database, from In each sampling time period in the verification cycle, t sampling time periods with the largest amount of query data are selected, wherein, the t is a natural number not less than 1; according to the hierarchical sampling algorithm, when the amount of query data is the largest Each sampling time period in the t sampling time periods takes M time points to sample the data of the source database and the target database queried by the core query script, wherein M is a natural number not less than N; M time points are sampled The sampled data of the source database and the target database are checked for data volume consistency, and the result of the data volume consistency check is updated once.

In this embodiment, if the data volume consistency check result after an update is that the data volumes of the sampled data sampled at M time points are consistent, then after the data volume consistency check result is updated once , also includes: obtaining the total query time corresponding to the data of the source database and the target database executed by the core query script in each sampling time period; calculating each sampling time according to the total query time corresponding to each sampling time period and the execution times of the core script The average time consumption and proportional time consumption of the segment.

Further, after calculating the average time-consuming and proportional time-consuming of each sampling time period according to the total query time and core script execution times corresponding to each sampling time period, it also includes: In the sampling time period, the t sampling time periods with the highest average time consumption and/or the highest proportional time consumption are selected; according to the stratified sampling algorithm, in the t sampling time periods with the highest average time consumption and/or proportional time consumption Each sampling period takes W time points to sample the data of the source database and the target database inquired by the core query script, wherein W is a natural number not less than N; The data is checked for data volume consistency, and the result of the data volume consistency check is updated a second time.

Specifically, in practical applications, directly performing data verification on sampling x sampling time periods may bring a relatively large load to the database. For example, the verification period is 1 hour, and the sampling time period is 5. If the data volume of these 5 time periods is directly verified, it means that the 12-minute data is directly taken out for comparison, and the data volume may be very large. It will lead to increased pressure on the database, slow down the speed of the entire verification process, and even affect online functions. Therefore, take a specific number of time points in each time period for statistics. When the data volume of these time points is consistent, then select more intensive and detailed time points for time periods with high data density and high database load. verify. Because the higher the data density or the greater the load, the lower the stability will be, so the possibility of data inconsistency will be greater.

In this embodiment, the data consistency check parameter also includes the number of retries; according to the preset hierarchical sampling algorithm, N time points are taken to sample the core query script query in each sampling time period The data of the source database and the target database, and perform data volume consistency verification. After obtaining the data volume consistency verification result, it also includes: if the data volume consistency verification result or the data volume consistency after an update If the result of the consistency check of the data volume after the calibration or secondary update is that the data volume is inconsistent, after a preset time delay, re-query the data of the source database and the target database according to the data consistency check parameters and the core query script, And the data volume consistency check is performed until the result of the data volume consistency check is that the data volume is consistent or the number of times of re-query reaches the number of retries.

Specifically, in this embodiment, a delayed retry mechanism is adopted to be compatible with scenarios such as shard collections. After the data volume consistency check result or the data volume consistency check after the first update or the data volume consistency check after the second update When the verification result shows that the amount of data is inconsistent, the delayed retry mechanism will be triggered until the number of delayed retries reaches the set number of retries.

After performing the data volume consistency check until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the number of retries, it also includes:

If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry count, a data consistency check result is obtained according to the data volume check result.

Specifically, if the data volume verification result is still inconsistent after repeated queries, the data consistency verification result will be generated directly according to the data volume verification result, without the need for subsequent data content consistency verification process, saving calculation resource.

104. If the data volume consistency verification result is that the data volume is consistent, perform data content consistency verification according to the data of the source database and the target database queried by the core query script, and obtain the data content consistency verification result;

In this embodiment, performing data content consistency verification on the data of the source database and the target database queried according to the core query script, and obtaining the result of the data content consistency verification include: according to the layered sampling algorithm , respectively sampling y pieces of data queried by the core query script in each sampling time period from the target database and the source database, wherein the y is a natural number not less than 1; The y pieces of data sampled in the source database are checked for data content consistency, and the result of the data content consistency check is obtained.

Further, the performing data content consistency verification on the y pieces of data sampled in the target database and the source database, and obtaining the result of the data content consistency verification include: performing the data content consistency verification on the target database and the source database respectively. Perform CRC encoding on the y pieces of data sampled in the source database to obtain corresponding encoding information; perform data content consistency verification on the encoding information corresponding to the source database and the target database, and obtain a data content consistency verification result.

Specifically, in practical applications, CRC is a cyclic redundancy check code, which is a commonly used check code in the field of computer network communication, and CRC encoding is performed through the cyclic redundancy check code. The cyclic redundancy check code includes a series of data coding rules such as shifting and division. Its algorithm principle, design and analysis of algorithm programs can be solved by corresponding software coding. CRC code is calculated according to the principle of modulo 2 division. Check code, and then attach the obtained check code to the data to be sent to form a new frame of data. After sending it to the receiving end, the receiving end recalculates the check code according to the rules of the generated check code. If they are the same, then It means that there is consistency in the data content, and if it is different, it means that there is no consistency in the data content.

In this embodiment, if the result of the data content consistency check is that the encoded information is consistent, the data content consistency check is performed on the y pieces of data sampled in the target database and the source database, After the data content consistency verification result is obtained, it also includes: performing data content consistency verification on the sampled data of the source database and the target database sampled at M time points, and updating the data content consistency verification result once ; If the data content consistency check result after an update is that the data content of the sampled data sampled at M time points is consistent, then the data content consistency check is performed on the sampled data of the source database and the target database sampled at W time points , and perform a secondary update on the result of the consistency check of the data content.

105. Obtain a data consistency verification result according to the data content consistency verification result.

In this embodiment, if the result of the data consistency verification is inconsistent, after the data consistency verification result is obtained according to the data content consistency verification result, it further includes: verifying the data consistency The result is recorded in the preset recording log, and the preset automatic execution script is executed, and the corresponding pocket compensation script is obtained according to the automatic execution script; according to the pocket compensation script, the sampling period of inconsistent data consistency check is obtained, And determine the data segment to be compensated in the target database according to the sampling time period when the data consistency check is inconsistent; obtain the data corresponding to the data segment to be compensated from the source database, and copy it to the target database.

Specifically, when the result of the data consistency check is that there is inconsistency in data volume or data content between the source database and the target database, the data in the target database that is inconsistent with the source database is compensated through the bottom compensation script, and the bottom compensation script It is a way of offline data compensation. According to the address cs of the pocket compensation script provided by the business party, the time range for which data compensation needs to be performed is passed in through a remote call, and the pocket compensation script will copy the data within this period to the target database, among which data compensation is required The time range is the sampling time period during which the comparison data is inconsistent during data volume consistency verification or data content consistency verification. Through this sampling time period, the data queried in the source database is determined and copied to the target database. Pocket compensation.

Specifically, after the data volume consistency check and data content consistency check are performed, if the obtained data consistency check result is consistent, it means that the data in the source database and the target database are consistent. At this time, release the data The resources occupied by volume consistency check and data content consistency check will be deleted, and the program will exit.

In this embodiment, by dividing the verification cycle according to the number of sampling time periods, the sampling time period is obtained, and the core query script of the execution times of the core script is executed in the sampling time period; according to the hierarchical sampling algorithm, at each sampling time Take N time points in the segment to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain the data volume consistency verification result; if the data volume consistency verification result is the data volume If they are consistent, the data content consistency verification is performed on the data of the source database and the target database queried by the core query script, and the result of the data content consistency verification is obtained. In this method, a specific number of time points are selected for sampling and verification in each time period through stratified sampling, thereby reducing the pressure on the database, and timely discovering data problems through periodic verification, improving the effectiveness of verification and reducing the pressure on the database.

The data consistency verification method in the embodiment of the present invention is described above, and the data consistency verification device in the embodiment of the present invention is described below, please refer to FIG. 2, an embodiment of the data consistency verification device in the embodiment of the present invention include:

The obtaining module 201 is used to obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles and sampling time periods, and the core query The script is used to query the data of the source database and the target database;

The query module 202 is used to divide the verification cycle according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the execution times of the core script in the sampling time period;

The data volume verification module 203 is used to take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script according to a preset hierarchical sampling algorithm, and perform data volume verification. Consistency verification, obtaining the result of data volume consistency verification, wherein the N is a natural number not less than 1;

The data content verification module 204 is configured to perform data content consistency verification according to the data of the source database and the target database queried by the core query script when the result of the data volume consistency verification is that the data volume is consistent, Obtain the result of data content consistency verification;

The verification result generating module 205 is configured to obtain a data consistency verification result according to the data content consistency verification result.

In the embodiment of the present invention, the data consistency verification device operates the above-mentioned data consistency verification method, and the data consistency verification device divides the verification period according to the number of sampling time periods to obtain the sampling time period, and The core query script that executes the number of times the core script is executed in the sampling period; according to the stratified sampling algorithm, take N time points in each sampling period to sample the data of the source database and the target database queried by the core query script, and calculate the data volume Consistency verification, to obtain the result of the consistency verification of the data volume; if the result of the consistency verification of the data volume is that the data volume is consistent, then the data content consistency verification is performed on the data of the source database and the target database queried by the core query script to obtain Data content consistency check result. In this method, a specific number of time points are selected for sampling and verification in each time period through stratified sampling, thereby reducing the pressure on the database, and timely discovering data problems through periodic verification, improving the effectiveness of verification and reducing the pressure on the database.

Referring to Fig. 3, the second embodiment of the data consistency checking device in the embodiment of the present invention includes:

The obtaining module 201 is used to obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles and sampling time periods, and the core query The script is used to query the data of the source database and the target database;

The query module 202 is used to divide the verification cycle according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the execution times of the core script in the sampling time period;

The data volume verification module 203 is used to take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script according to a preset hierarchical sampling algorithm, and perform data volume verification. Consistency verification, obtaining the result of data volume consistency verification, wherein the N is a natural number not less than 1;

The data content verification module 204 is configured to perform data content consistency verification according to the data of the source database and the target database queried by the core query script when the result of the data volume consistency verification is that the data volume is consistent, Obtain the result of data content consistency verification;

The verification result generating module 205 is configured to obtain a data consistency verification result according to the data content consistency verification result.

In this embodiment, the data consistency checking device further includes a first updating module 206, and the first updating module 206 is specifically used for:

According to the query data volume of the core query script that executes the core script execution times in each sampling time period to query the data of the source database and the target database, filter out the largest query data volume from each sampling time period in the verification cycle t sampling time periods, wherein the t is a natural number not less than 1;

According to the hierarchical sampling algorithm, M time points are taken to sample the data of the source database and the target database queried by the core query script in each of the t sampling time periods with the largest amount of query data, wherein M is a natural number not less than N;

Perform a data volume consistency check on the sampled data of the source database and the target database sampled at M time points, and update the data volume consistency check result once.

In this embodiment, the data consistency checking device further includes a time-consuming calculation module 207, and the time-consuming calculation module 207 is specifically used for:

Obtain the total query time corresponding to the data in the source database and the target database by executing the core query script in each sampling time period;

According to the total query time and core script execution times corresponding to each sampling time period, the average time consumption and proportional time consumption of each sampling time period are calculated.

In this embodiment, the data consistency checking device further includes a second updating module 208, and the second updating module 208 is specifically used for:

Selecting the t sampling time periods with the highest average time-consuming and/or proportional time-consuming from each sampling time period of the verification cycle;

According to the hierarchical sampling algorithm, W time points are taken in each of the t sampling time periods with the highest average time-consuming and/or proportional time-consuming to sample the source database and target database queried by the core query script data, where W is a natural number not less than N;

The data volume consistency check is performed on the sampled data of the source database and the target database sampled at W time points, and a second update is performed on the data volume consistency check result.

In this embodiment, the data consistency check parameter also includes the number of retries; the data consistency check device also includes a retry module 209, and the retry module 209 is specifically used for:

If the data volume consistency check result or the data volume consistency check after the first update or the data volume consistency check result after the second update is that the data volume is inconsistent, after a preset time delay, according to the data Consistency check parameters and core query scripts re-query the data of the source database and the target database, and perform data volume consistency checks until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the retry frequency.

In this embodiment, the retry module 209 is specifically further configured to:

If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry count, a data consistency check result is obtained according to the data volume check result.

In this embodiment, the data content checking module 204 is specifically used for:

The data sampling unit 2041 is configured to sample y pieces of data queried by the core query script in each sampling time period from the target database and the source database according to the hierarchical sampling algorithm, wherein the y is A natural number not less than 1;

The content consistency verification unit 2042 is configured to perform data content consistency verification on the y pieces of data sampled in the target database and the source database, and obtain a data content consistency verification result.

In this embodiment, the content consistency checking unit 2042 is specifically used for:

Performing CRC encoding on the y pieces of data sampled in the target database and the source database respectively, to obtain corresponding encoding information;

Performing a data content consistency check on the encoding information corresponding to the source database and the target database, to obtain a data content consistency check result.

In this embodiment, the data consistency checking device further includes a third updating module 210, and the third updating module 210 is specifically used for:

Performing a data content consistency check on the sampled data of the source database and the target database sampled at M time points, and updating the result of the data content consistency check once;

If the data content consistency check result after an update is that the data content of the sampled data sampled at M time points is consistent, then the data content consistency check is performed on the sampled data of the source database and the target database sampled at W time points, And a second update is performed on the result of the consistency check of the data content.

In this embodiment, the data consistency verification device further includes a bottom compensation module 211, and the retry module 211 is specifically used for:

Recording the data consistency verification result in a preset recording log, and executing a preset automatic execution script, and obtaining a corresponding bottom-up compensation script according to the automatic execution script;

Acquiring the sampling time period during which the data consistency check is inconsistent according to the bottom-up compensation script, and determining the data segment to be compensated in the target database according to the sampling time period during which the data consistency check is inconsistent;

The data corresponding to the data segment to be compensated is obtained from the source database and copied to the target database.

On the basis of the previous embodiment, this embodiment describes in detail the specific functions of each module and the unit composition of some modules. By dividing the verification cycle according to the number of sampling time periods through the above modules and module units, the sampling time period is obtained. And execute the core query script of the core script execution times in the sampling time period; according to the hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform Data volume consistency check to get the data volume consistency check result; if the data volume consistency check result is the same data volume, perform data content consistency check on the data in the source database and the target database queried by the core query script , to get the result of the consistency check of the data content. In this method, a specific number of time points are selected for sampling and verification in each time period through stratified sampling, thereby reducing the pressure on the database, and timely discovering data problems through periodic verification, improving the effectiveness of verification and reducing the pressure on the database.

The above Figures 2 and 3 describe in detail the data consistency verification device in the embodiment of the present invention from the perspective of modular functional entities, and the following describes the data consistency verification device in the embodiment of the present invention in detail from the perspective of hardware processing .

FIG. 4 is a schematic structural diagram of a data consistency verification device provided by an embodiment of the present invention. The data consistency verification device 400 may have relatively large differences due to different configurations or performances, and may include one or more processors. (central processing units, CPU) 410 (for example, one or more processors) and memory 420, and one or more storage media 430 for storing application programs 433 or data 432 (for example, one or more mass storage devices). Wherein, the memory 420 and the storage medium 430 may be temporary storage or persistent storage. The program stored in the storage medium 430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations for the data consistency checking device 400 . Furthermore, the processor 410 may be configured to communicate with the storage medium 430, and execute a series of instruction operations in the storage medium 430 on the data consistency verification device 400, so as to implement the following steps:

Obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles, and sampling time periods, and the core query scripts are used to query the source database and target database data;

Dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the execution times of the core script in the sampling time period;

According to the preset hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain consistent data volume A sex check result, wherein said N is a natural number not less than 1;

If the data volume consistency verification result is that the data volume is consistent, then perform data content consistency verification according to the data of the source database and the target database queried by the core query script, and obtain a data content consistency verification result;

A data consistency verification result is obtained according to the data content consistency verification result.

Optionally, according to the preset stratified sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency Verification, after obtaining the data volume consistency verification result, also includes:

According to the query data volume of the core query script that executes the core script execution times in each sampling time period to query the data of the source database and the target database, filter out the largest query data volume from each sampling time period in the verification cycle t sampling time periods, wherein the t is a natural number not less than 1;

According to the hierarchical sampling algorithm, M time points are taken to sample the data of the source database and the target database queried by the core query script in each of the t sampling time periods with the largest amount of query data, wherein M is a natural number not less than N;

Perform a data volume consistency check on the sampled data of the source database and the target database sampled at M time points, and update the data volume consistency check result once.

Optionally, if the data volume consistency check result after an update is that the data volumes of the sampled data sampled at M time points are consistent, then after the data volume consistency check result is updated once, further include:

Obtain the total query time corresponding to the data in the source database and the target database by executing the core query script in each sampling time period;

According to the total query time and core script execution times corresponding to each sampling time period, the average time consumption and proportional time consumption of each sampling time period are calculated.

Optionally, after calculating the average time-consuming and proportional time-consuming of each sampling time period according to the total query time and core script execution times corresponding to each sampling time period, the method further includes:

Selecting the t sampling time periods with the highest average time-consuming and/or proportional time-consuming from each sampling time period of the verification cycle;

According to the hierarchical sampling algorithm, W time points are taken in each of the t sampling time periods with the highest average time-consuming and/or proportional time-consuming to sample the source database and target database queried by the core query script data, where W is a natural number not less than N;

The data volume consistency check is performed on the sampled data of the source database and the target database sampled at W time points, and a second update is performed on the data volume consistency check result.

Optionally, the data consistency check parameter also includes the number of retries;

According to the preset layered sampling algorithm, N time points are taken to sample the data of the source database and the target database queried by the core query script in each sampling time period, and the data volume consistency check is performed to obtain After the data volume consistency check result, it also includes:

If the data volume consistency check result or the data volume consistency check after the first update or the data volume consistency check result after the second update is that the data volume is inconsistent, after a preset time delay, according to the data Consistency check parameters and core query scripts re-query the data of the source database and the target database, and perform data volume consistency checks until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the retry frequency.

Optionally, after the data volume consistency check is performed until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the number of retries, the method further includes:

If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry count, a data consistency check result is obtained according to the data volume check result.

Optionally, performing the data content consistency check on the data of the source database and the target database queried according to the core query script, and obtaining the result of the data content consistency check include:

According to the hierarchical sampling algorithm, sample y pieces of data queried by the core query script in each sampling time period from the target database and the source database respectively, wherein the y is a natural number not less than 1;

Performing a data content consistency check on the y pieces of data sampled in the target database and the source database to obtain a data content consistency check result.

Optionally, performing data content consistency verification on the y pieces of data sampled in the target database and the source database, and obtaining a data content consistency verification result include:

Performing CRC encoding on the y pieces of data sampled in the target database and the source database respectively, to obtain corresponding encoding information;

Performing a data content consistency check on the encoding information corresponding to the source database and the target database, to obtain a data content consistency check result.

Optionally, if the result of the data content consistency check is that the encoded information is consistent, then perform a data content consistency check on the y pieces of data sampled in the target database and the source database to obtain data After the content consistency verification results, it also includes:

Performing a data content consistency check on the sampled data of the source database and the target database sampled at M time points, and updating the result of the data content consistency check once;

If the data content consistency check result after an update is that the data content of the sampled data sampled at M time points is consistent, then the data content consistency check is performed on the sampled data of the source database and the target database sampled at W time points, And a second update is performed on the result of the consistency check of the data content.

Optionally, if the result of the data consistency verification is inconsistent, after the data consistency verification result is obtained according to the data content consistency verification result, the method further includes:

Recording the data consistency verification result in a preset recording log, and executing a preset automatic execution script, and obtaining a corresponding bottom-up compensation script according to the automatic execution script;

Acquiring the sampling time period during which the data consistency check is inconsistent according to the bottom-up compensation script, and determining the data segment to be compensated in the target database according to the sampling time period during which the data consistency check is inconsistent;

The data corresponding to the data segment to be compensated is obtained from the source database and copied to the target database.

The data consistency checking device 400 may also include one or more power supplies 440, one or more wired or wireless network interfaces 450, one or more input and output interfaces 460, and/or, one or more operating systems 431, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and more. Those skilled in the art can understand that the structure of the data consistency verification device shown in Figure 4 does not constitute a limitation on the data consistency verification device provided by the present invention, and may include more or less components than those shown in the figure, or combine certain components, or a different arrangement of components.

The present invention also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium may also be a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium, and when the instructions are run on the computer, the computer is made to perform the following steps:

Obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles, and sampling time periods, and the core query scripts are used to query the source database and target database data;

Dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the execution times of the core script in the sampling time period;

According to the preset hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain consistent data volume A sex check result, wherein said N is a natural number not less than 1;

If the data volume consistency verification result is that the data volume is consistent, then perform data content consistency verification according to the data of the source database and the target database queried by the core query script, and obtain a data content consistency verification result;

A data consistency verification result is obtained according to the data content consistency verification result.

Optionally, according to the preset stratified sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency Verification, after obtaining the data volume consistency verification result, also includes:

According to the query data volume of the core query script that executes the core script execution times in each sampling time period to query the data of the source database and the target database, filter out the largest query data volume from each sampling time period in the verification cycle t sampling time periods, wherein the t is a natural number not less than 1;

According to the hierarchical sampling algorithm, M time points are taken to sample the data of the source database and the target database queried by the core query script in each of the t sampling time periods with the largest amount of query data, wherein M is a natural number not less than N;

Perform a data volume consistency check on the sampled data of the source database and the target database sampled at M time points, and update the data volume consistency check result once.

Optionally, if the data volume consistency check result after an update is that the data volumes of the sampled data sampled at M time points are consistent, then after the data volume consistency check result is updated once, further include:

Obtain the total query time corresponding to the data in the source database and the target database by executing the core query script in each sampling time period;

According to the total query time and core script execution times corresponding to each sampling time period, the average time consumption and proportional time consumption of each sampling time period are calculated.

Optionally, after calculating the average time-consuming and proportional time-consuming of each sampling time period according to the total query time and core script execution times corresponding to each sampling time period, the method further includes:

Selecting the t sampling time periods with the highest average time-consuming and/or proportional time-consuming from each sampling time period of the verification cycle;

According to the hierarchical sampling algorithm, W time points are taken in each of the t sampling time periods with the highest average time-consuming and/or proportional time-consuming to sample the source database and target database queried by the core query script data, where W is a natural number not less than N;

The data volume consistency check is performed on the sampled data of the source database and the target database sampled at W time points, and a second update is performed on the data volume consistency check result.

Optionally, the data consistency check parameter also includes the number of retries;

According to the preset layered sampling algorithm, N time points are taken to sample the data of the source database and the target database queried by the core query script in each sampling time period, and the data volume consistency check is performed to obtain After the data volume consistency check result, it also includes:

If the data volume consistency check result or the data volume consistency check after the first update or the data volume consistency check result after the second update is that the data volume is inconsistent, after a preset time delay, according to the data Consistency check parameters and core query scripts re-query the data of the source database and the target database, and perform data volume consistency checks until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the retry frequency.

Optionally, after the data volume consistency check is performed until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the number of retries, the method further includes:

If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry count, a data consistency check result is obtained according to the data volume check result.

Optionally, performing the data content consistency check on the data of the source database and the target database queried according to the core query script, and obtaining the result of the data content consistency check include:

According to the hierarchical sampling algorithm, sample y pieces of data queried by the core query script in each sampling time period from the target database and the source database respectively, wherein the y is a natural number not less than 1;

Performing a data content consistency check on the y pieces of data sampled in the target database and the source database to obtain a data content consistency check result.

Optionally, performing data content consistency verification on the y pieces of data sampled in the target database and the source database, and obtaining a data content consistency verification result include:

Performing CRC encoding on the y pieces of data sampled in the target database and the source database respectively, to obtain corresponding encoding information;

Performing a data content consistency check on the encoding information corresponding to the source database and the target database, to obtain a data content consistency check result.

Optionally, if the result of the data content consistency check is that the encoded information is consistent, then perform a data content consistency check on the y pieces of data sampled in the target database and the source database to obtain data After the content consistency verification results, it also includes:

Performing a data content consistency check on the sampled data of the source database and the target database sampled at M time points, and updating the result of the data content consistency check once;

If the data content consistency check result after an update is that the data content of the sampled data sampled at M time points is consistent, then the data content consistency check is performed on the sampled data of the source database and the target database sampled at W time points, And a second update is performed on the result of the consistency check of the data content.

Optionally, if the result of the data consistency verification is inconsistent, after the data consistency verification result is obtained according to the data content consistency verification result, the method further includes:

Recording the data consistency verification result in a preset recording log, and executing a preset automatic execution script, and obtaining a corresponding bottom-up compensation script according to the automatic execution script;

Acquiring the sampling time period during which the data consistency check is inconsistent according to the bottom-up compensation script, and determining the data segment to be compensated in the target database according to the sampling time period during which the data consistency check is inconsistent;

The data corresponding to the data segment to be compensated is obtained from the source database and copied to the target database.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system, device, and unit described above can refer to the corresponding process in the foregoing method embodiments, and details are not repeated here.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions recorded in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (13)

1. A data consistency verification method, characterized in that, the data consistency verification method comprises: Obtain preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include the number of core script execution times, verification cycles, and sampling time periods, and the core query scripts are used to query the source database and target database data; Dividing the verification period according to the number of sampling time periods to obtain a sampling time period, and querying the core query script according to the execution times of the core script in the sampling time period; According to the preset hierarchical sampling algorithm, take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script, and perform data volume consistency verification to obtain consistent data volume A sex check result, wherein said N is a natural number not less than 1; If the data volume consistency verification result is that the data volume is consistent, then perform data content consistency verification according to the data of the source database and the target database queried by the core query script, and obtain a data content consistency verification result; A data consistency verification result is obtained according to the data content consistency verification result. 2. The data consistency verification method according to claim 1, wherein, according to the preset layered sampling algorithm, N time points are taken to sample the core query script in each sampling period Query the data of the source database and the target database, and perform data volume consistency verification. After obtaining the data volume consistency verification results, it also includes: According to the query data volume of the core query script that executes the core script execution times in each sampling time period to query the data of the source database and the target database, filter out the largest query data volume from each sampling time period in the verification cycle t sampling time periods, wherein the t is a natural number not less than 1; According to the hierarchical sampling algorithm, M time points are taken to sample the data of the source database and the target database queried by the core query script in each of the t sampling time periods with the largest amount of query data, wherein M is a natural number not less than N; Perform a data volume consistency check on the sampled data of the source database and the target database sampled at M time points, and update the data volume consistency check result once. 3. The data consistency check method according to claim 2, wherein if the data volume consistency check result after an update is that the data volumes of the sampling data sampled at M time points are consistent, then in the After the data volume consistency check result is updated once, it also includes: Obtain the total query time corresponding to the data in the source database and the target database by executing the core query script in each sampling time period; According to the total query time and core script execution times corresponding to each sampling time period, the average time consumption and proportional time consumption of each sampling time period are calculated. 4. The data consistency verification method according to claim 3, wherein, in the said query total time corresponding to each sampling time period and the execution times of core scripts, calculate the average time-consuming of each sampling time period And after proportional time-consuming, also include: Selecting the t sampling time periods with the highest average time-consuming and/or proportional time-consuming from each sampling time period of the verification cycle; According to the hierarchical sampling algorithm, W time points are taken in each of the t sampling time periods with the highest average time-consuming and/or proportional time-consuming to sample the source database and target database queried by the core query script data, where W is a natural number not less than N; The data volume consistency check is performed on the sampled data of the source database and the target database sampled at W time points, and a second update is performed on the data volume consistency check result. 5. The data consistency verification method according to claim 4, wherein the data consistency verification parameter also includes the number of retries; According to the preset layered sampling algorithm, N time points are taken to sample the data of the source database and the target database queried by the core query script in each sampling time period, and the data volume consistency check is performed to obtain After the data volume consistency check result, it also includes: If the data volume consistency check result or the data volume consistency check after the first update or the data volume consistency check result after the second update is that the data volume is inconsistent, after a preset time delay, according to the data Consistency check parameters and core query scripts re-query the data of the source database and the target database, and perform data volume consistency checks until the result of the data volume consistency check is that the data volume is consistent or the number of re-queries reaches the retry frequency. 6. The data consistency verification method according to claim 5, wherein, performing the data volume consistency verification until the data volume consistency verification result is that the data volume is consistent or the number of times of re-query reaches the specified number of times. After the number of retries stated, also include: If the data volume consistency check result is inconsistent after the number of re-queries reaches the retry count, a data consistency check result is obtained according to the data volume check result. 7. The data consistency check method according to claim 3, wherein the data of the source database and the target database queried according to the core query script are checked for data content consistency to obtain data content Consistency check results include: According to the hierarchical sampling algorithm, sample y pieces of data queried by the core query script in each sampling time period from the target database and the source database respectively, wherein the y is a natural number not less than 1; Performing a data content consistency check on the y pieces of data sampled in the target database and the source database to obtain a data content consistency check result. 8. The method for checking data consistency according to claim 7, wherein the y pieces of data sampled in the target database and the source database are checked for consistency of data content to obtain data content Consistency check results include: Performing CRC encoding on the y pieces of data sampled in the target database and the source database respectively, to obtain corresponding encoding information; Performing a data content consistency check on the encoding information corresponding to the source database and the target database, to obtain a data content consistency check result. 9. The data consistency check method according to claim 8, wherein if the result of the data content consistency check is that the coded information is consistent, then the target database and the source database are performed on the pair. The y pieces of data sampled in are checked for data content consistency, and after the data content consistency check result is obtained, it also includes: Performing a data content consistency check on the sampled data of the source database and the target database sampled at M time points, and updating the result of the data content consistency check once; If the data content consistency check result after an update is that the data content of the sampled data sampled at M time points is consistent, then the data content consistency check is performed on the sampled data of the source database and the target database sampled at W time points, And a second update is performed on the result of the consistency check of the data content. 10. The data consistency verification method according to claim 1, wherein if the data consistency verification result is inconsistent, then the data consistency verification result is obtained according to the data content consistency verification result. After the results, also include: Recording the data consistency verification result in a preset recording log, and executing a preset automatic execution script, and obtaining a corresponding bottom-up compensation script according to the automatic execution script; Acquiring the sampling time period during which the data consistency check is inconsistent according to the bottom-up compensation script, and determining the data segment to be compensated in the target database according to the sampling time period during which the data consistency check is inconsistent; The data corresponding to the data segment to be compensated is obtained from the source database and copied to the target database. 11. A data consistency verification device, characterized in that, the data consistency verification device comprises: An acquisition module, configured to acquire preset data consistency verification parameters and core query scripts, wherein the data consistency verification parameters include core script execution times, verification cycles and the number of sampling time periods, and the core query scripts Data used to query the source and target databases; A query module, configured to divide the verification cycle according to the number of sampling time periods to obtain a sampling time period, and query the core query script according to the execution times of the core script in the sampling time period; The data volume verification module is used to take N time points in each sampling time period to sample the data of the source database and the target database queried by the core query script according to a preset layered sampling algorithm, and perform data volume consistency Consistency verification, to obtain the consistency verification result of the data volume, wherein, the N is a natural number not less than 1; A data content verification module, configured to perform a data content consistency verification according to the data of the source database and the target database queried by the core query script when the result of the data volume consistency verification is that the data volume is consistent, to obtain Data content consistency check result; A verification result generating module, configured to obtain a data consistency verification result according to the data content consistency verification result. 12. A data consistency verification device, characterized in that the data consistency verification device comprises: a memory and at least one processor, and instructions are stored in the memory; The at least one processor invokes the instruction in the memory, so that the data consistency checking device executes the steps of the data consistency checking method according to any one of claims 1-10. 13. A computer-readable storage medium, wherein instructions are stored on the computer-readable storage medium, wherein when the instructions are executed by a processor, the data consistency as described in any one of claims 1-10 is realized The steps of the verification method.

Images