CN109885614B - Data synchronization method and device - Google Patents
Data synchronization method and device Download PDFInfo
- Publication number
- CN109885614B CN109885614B CN201910008352.4A CN201910008352A CN109885614B CN 109885614 B CN109885614 B CN 109885614B CN 201910008352 A CN201910008352 A CN 201910008352A CN 109885614 B CN109885614 B CN 109885614B
- Authority
- CN
- China
- Prior art keywords
- data table
- data
- field
- target data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000001360 synchronised effect Effects 0.000 claims abstract description 73
- 238000003780 insertion Methods 0.000 claims abstract description 23
- 230000037431 insertion Effects 0.000 claims abstract description 23
- 238000004590 computer program Methods 0.000 claims description 21
- 238000013479 data entry Methods 0.000 claims description 18
- 238000013524 data verification Methods 0.000 claims description 8
- 238000012795 verification Methods 0.000 claims description 7
- 238000012545 processing Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013496 data integrity verification Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention is suitable for the technical field of data processing, and provides a method and a device for data synchronization, wherein the method comprises the following steps: configuring a corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises a data table corresponding relation between the source data table and the target data table and a field corresponding relation between each field in the source data table and each field in the target data table; determining the maximum value of the number of the data item in the target data table; generating a data synchronous execution statement according to the maximum value of the data item numbers in the target data table and the corresponding relation between the data table and the field, wherein the data synchronous execution statement comprises a query statement and an insertion statement; executing the query statement to extract the data to be synchronized from the source data table; executing an insertion statement to insert the data to be synchronized into the target data table; and carrying out integrity check on the target data table. The embodiment can realize increment synchronization between the source data table and the target data table.
Description
Technical Field
The present invention relates to a data synchronization method, a data synchronization device, a device and a computer readable storage medium, and more particularly, to a data synchronization method, a data synchronization device, a device and a computer readable storage medium.
Background
The data synchronization may synchronize data in a certain data table to other data tables. Currently, synchronization of data between databases is mainly achieved through full-scale synchronization. Full synchronization refers to synchronizing all data in a source data table together to a target data table. In general, the number of data tables stored in each data table is very huge, if a full-scale synchronization mode is adopted, each synchronization consumes very much resources, and the time consumption is relatively long, so that the normal use of the data table or the database is affected.
On the other hand, in order to ensure the consistency of the data before and after synchronization, the integrity check of the data after synchronization needs to be performed regularly. If the full-volume synchronization mode is adopted, because the full-volume synchronization is to synchronize all data in the source data table to the target data table together, the data in the target data table can only be directly compared with the data in the source data table when the integrity of the data is checked, and the check accuracy is lower due to lack of comparison.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a method and an apparatus for data synchronization, so as to solve the problems in the prior art that the data synchronization resource consumption is high, the time consumption is long, and the accuracy of data verification after synchronization is low by adopting full synchronization.
A first aspect of an embodiment of the present invention provides a method for data synchronization, including:
configuring a corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises a data table corresponding relation between the source data table and the target data table and a field corresponding relation between each field in the source data table and each field in the target data table;
Determining the maximum value of the number of the data item in the target data table;
generating a data synchronous execution statement according to the maximum value of the data item numbers in the target data table and the corresponding relation between the data table and the field, wherein the data synchronous execution statement comprises a query statement and an insertion statement;
Executing the query statement to extract data to be synchronized from the source data table;
Executing the insertion statement to insert the data to be synchronized into the target data table;
and carrying out integrity check on the target data table.
A second aspect of an embodiment of the present invention provides an apparatus for data synchronization, including:
The corresponding relation configuration module is used for configuring the corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises the data table corresponding relation between the source data table and the target data table and the field corresponding relation between each field in the source data table and each field in the target data table;
an entry number determining module, configured to determine a maximum value of the data entry numbers in the target data table;
The execution statement generation module is used for generating a data synchronous execution statement according to the maximum value of the number of the data item in the target data table and the corresponding relation between the data table and the field, wherein the data synchronous execution statement comprises a query statement and an insertion statement;
The data to be synchronized query module is used for executing the query statement to extract data to be synchronized from the source data table;
The data to be synchronized is inserted into the target data table by executing the insertion statement;
and the data verification module is used for carrying out integrity verification on the target data table.
A third aspect of the embodiments of the present invention provides an apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method of data synchronization described above when executing the computer program.
A fourth aspect of the embodiments of the present invention provides a computer readable storage medium storing a computer program which, when executed by a processor, performs the steps of the method of data synchronization described above.
Compared with the prior art, the embodiment of the invention has the following advantages:
According to the embodiment of the invention, the maximum value of the number of the data item in the target data table is determined by configuring the corresponding relation between the source data table and the target data table, so that the data synchronous execution statement comprising the query statement and the insertion statement can be generated according to the maximum value of the number of the data item in the target data table and the corresponding relation between the source data table and the target data table and the corresponding relation between the data table and the fields. The data to be synchronized can be extracted from the source data table by executing the query statement, the data to be synchronized can be inserted into the target data table by executing the insertion statement, and the integrity of the target data table is checked. The embodiment can realize increment synchronization between the source data table and the target data table, so that only newly added contents in the source data table can be synchronized during data synchronization, the occupation of the data synchronization on resources is reduced, and the efficiency of the data synchronization is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings that are required to be used in the embodiments or the description of the prior art. It is evident that the figures in the following description are only some embodiments of the invention, from which other figures can be obtained without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart illustrating steps of a method for data synchronization according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating steps of another method of data synchronization according to one embodiment of the present invention;
FIG. 3 is a schematic diagram of an apparatus for data synchronization in accordance with one embodiment of the present invention;
Fig. 4 is a schematic diagram of an apparatus according to one embodiment of the invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
The technical scheme of the invention is described below through specific examples.
Referring to fig. 1, a flowchart illustrating steps of a method for data synchronization according to an embodiment of the present invention may specifically include the following steps:
s101, configuring a corresponding relation between a source data table and a target data table;
In the embodiment of the present invention, the source data table and the target data table may be data tables stored in different databases, or may be data tables stored in the same database, which is not limited in this embodiment.
For easy understanding, the present embodiment will be described and explained below by taking the example that the source data table and the target data table are stored in different databases. That is, it can be considered that the source data table is stored in a first database and the target data table is stored in a second database, and the first database and the second database are different databases.
In addition, the target data table in this embodiment may be an empty table, that is, no other data is stored; alternatively, the target data table may be a data table storing other data, which is not limited in this embodiment.
In the embodiment of the invention, before data synchronization is performed, the corresponding relationship between the source data table and the target data table can be configured first. The corresponding relation not only comprises the data table corresponding relation between the source data table and the target data table, but also comprises the field corresponding relation between each field in the source data table and each field in the target data table.
For example, there are two data tables, namely, a data table a and a data table B, and when the data in the data table a needs to be synchronized to the data table B, the data table a is taken as a source data table, the data table B is taken as a target data table, and the data table correspondence between the data table a and the data table B can be established first.
Then, the field correspondence between each field in the data table a and the data table B is further configured. Assuming that there are fields Aa, ab and Ac in the data table a, and fields Ba, bb and Bc in the data table B, it is now necessary to synchronize the data stored under the field Aa in the data table a to the field Ba in the data table B, synchronize the data stored under the field Ab in the data table a to the field Bb in the data table B, and synchronize the data stored under the field Ac in the data table a to the field Bc in the data table B, then the correspondence relationship between the three fields may be established, as shown in the following table:
Table I, example of correspondence between source data Table and target data Table
It should be noted that, when the correspondence between the fields is configured, the correspondence between all the fields stored in the source data table and a certain field in the target data table may be established, or the correspondence between only the field in the source data table, which needs to be data synchronized, and a certain field in the target data table may be established, which is not limited in this embodiment.
For example, assuming that there are fields Aa, ab, ac and Ad in the data table a, but only data stored under three fields Aa, ab, ac need to be synchronized during data synchronization, only the three fields Aa, ab, ac may be respectively associated with a certain field in the target data table, or the four fields Aa, ab, ac and Ad may be respectively associated with a certain field in the target data table.
After the corresponding relation between the source data table and the target data table and the corresponding relation between the fields in the two data tables are configured, the corresponding relation can be written into a configuration file for storage.
S102, determining the maximum value of the number of the data item in the target data table;
Typically, the fields in the data table are stored in an order that can be represented by the self-increment ID. The self-increment ID in the data table may be an entry number of the data store in the data table. The data entry number is unique for each piece of data. For example, the data entry numbers in the source data table may be recorded as 1,2, 3 … …, and correspondingly, the data entry numbers in the target data table may be displayed as 1,2, 3 … …, respectively.
In the embodiment of the invention, which data in the source data table is synchronized into the target data table can be found by determining the maximum value of the data item numbers in the target data table, so that only the data generated after the item numbers can be synchronized, incremental synchronization of the data table is realized, all the data in the source data table are not needed to be synchronized again, and the synchronous data volume is reduced.
S103, generating a data synchronous execution statement according to the maximum value of the data item numbers in the target data table and the corresponding relation between the data table and the field;
in embodiments of the present invention, data synchronization may be achieved by executing a particular SQL (Structured Query Language ) statement.
The SQL statement may include a query statement that looks up data in the source data table and an insert statement that inserts the looked up data into the target data table.
In a specific implementation, the operator may generate the query statement and the insert statement by way of script code.
For example, the query statement may be represented as "SELECT FIELD AA from table A", i.e., data under field Aa is selected from the source data table A; and the insert statement may be denoted as "insert into table B", i.e., insert the data under field Aa in source data table a into destination data table B.
Of course, the above statement is only an example, and a person skilled in the art can write appropriate query statements and insert statements according to actual needs. For example, more specific constraints may be added to the SQL statement, such as looking up specific fields, data generated during a certain period of time, etc., to achieve personalized data synchronization, etc.
In the embodiment of the invention, the field to be synchronized in the source data table can be determined first, the query statement for the source data table is generated according to the field to be synchronized in the source data table and the maximum value of the number of the data item in the target data table, and then the insertion statement for the target data table is generated according to the corresponding relationship of the data table and the corresponding relationship of the field.
In a specific implementation, the maximum value of the data entry number of the synchronized data in the target data table can be determined according to the corresponding relation between the source data table and the target data table, and then the maximum value of the data entry number is added into the SQL sentence, so that only data larger than the maximum value of the data entry number is searched.
For example, in the above example, the maximum value of the data entry number in the target data table is 1000, and then when the SQL statement for incremental data synchronization is generated, the maximum value of the data entry number of 1000 may be added to the query statement, that is, the query statement of "SELECT FIELD AA over 1000 from table A" is used to find out the data with the data entry number greater than 1000 from the source data table.
Then, an insert sentence "insert into field Ba in table B" for the target data table is generated according to the data table correspondence and the field correspondence.
By adding the variable of the data entry number in the SQL statement for executing the synchronization operation, the incremental synchronization of the incremental data in the source data table can be realized without performing full synchronization on all the data at one time.
S104, executing the query statement to extract data to be synchronized from the source data table;
In the embodiment of the invention, the complete process of completing data synchronization by executing SQL sentences comprises the steps of searching the data to be synchronized from a source data table and inserting the data to be synchronized into a target data table.
Thus, a query statement, such as "SELECT FIELD AA from table a", that looks up the data to be synchronized from the source data table may be first executed, the data stored under field Aa is extracted from the source data table a, and then an insert statement, such as "insert into table B", that inserts the data to be synchronized into the target data table is executed.
Note that the data synchronization in this embodiment may be synchronization of incremental data, and thus the frequency of synchronization may be set relatively high. For example, it may be arranged to perform a sync SQL statement every 10 minutes to synchronize incremental data.
In synchronizing incremental data, this may be accomplished by executing an SQL statement that contains the data entry numbers of the data tables.
For example, by executing the query statement "SELECT FIELD AA over 1000 from table A", data having a data entry number greater than 1000 may be found from the source data table, and then the found data may be synchronized to the target data table.
S105, executing the insertion statement to insert the data to be synchronized into the target data table;
for example, an insert statement of "insert into field Ba in table B" may be executed to insert data having a data entry number greater than 1000 found in the source data table into field Ba of the destination data table.
S106, carrying out integrity check on the target data table.
Data integrity verification is a verification operation performed to ensure the integrity and accuracy of data. In the embodiment of the invention, the data integrity check can verify whether the synchronized data is the same as the source data.
In the embodiment of the invention, the source data table can be used as a reference, and the synchronization of the full data is completed once at first, namely, all the data in the source data table A are synchronized into the check data table C.
Since the full-volume synchronization occupies more resources and takes longer time, the full-volume synchronization can be performed in idle time, such as early morning.
After the full-scale synchronization is completed, the data in the check data table C can be compared with the target data table B obtained after the previous synchronization to verify the integrity and accuracy of the data.
In the embodiment of the invention, the data synchronous execution statement comprising the query statement and the insertion statement can be generated according to the maximum value of the number of the data item in the target data table and the data table corresponding relation and the field corresponding relation between the source data table and the target data table by configuring the corresponding relation between the source data table and the target data table and determining the maximum value of the number of the data item in the target data table. The data to be synchronized can be extracted from the source data table by executing the query statement, the data to be synchronized can be inserted into the target data table by executing the insertion statement, and the integrity of the target data table is checked. The embodiment can realize increment synchronization between the source data table and the target data table, so that only newly added contents in the source data table can be synchronized during data synchronization, the occupation of the data synchronization on resources is reduced, and the efficiency of the data synchronization is improved.
Referring to fig. 2, a flowchart illustrating steps of another method for data synchronization according to an embodiment of the present invention may specifically include the following steps:
s201, determining dynamic codes of all fields in the source data table;
in an embodiment of the present invention, the source data table may be a data table having multiple attribute fields. The data table with multiple attribute fields means that the field attributes corresponding to each piece of data recorded in the same data table comprise multiple kinds.
For example, in a certain source data table, 4 pieces of data are recorded as follows:
table two, multi-Attribute field data Table example
| Field 1 | Field 2 | Field 3 | |
| Data 1 | Clothing 1 | Color of | Red colour |
| Data of item 2 | Clothes 2 | Color of | Green, green |
| Data 3 rd | Clothing 1 | Size of the article | 38 |
| Data of item 4 | Clothes 2 | Size of the article | 40 |
Two different attributes of clothes, namely color and size, are recorded in the data table, and attribute values corresponding to the attributes are recorded in the source data table in the form of characters (Chinese characters or English characters).
As can be seen from the example of table two above, the data table repeatedly records both color and size attributes in the form of multiple fields, resulting in redundancy of data. By the data synchronization method in the present embodiment, these data redundancies can be reduced.
In data synchronization of the data table, the dynamic code (code) of each field in the source data table may be first determined.
S202, associating the data in each field according to the dynamic codes so that the data with the same ID corresponds to the same dynamic code;
In the embodiment of the invention, the data under a certain field can be used as a reference, and the data under the field with the same type can be corresponding to the same dynamic code.
For example, with respect to the data in table two, the dynamic code may be determined based on the data recorded under field 1, based on the clothing ID recorded in field 1, so that pieces of data having the same clothing ID have the same dynamic code.
Therefore, in the data record of the source data table, the dynamic codes of the 1 st data and the 3 rd data are the same, and may be represented as code1, the dynamic codes of the 2 nd data and the 4 th data are the same, and represented as code2, so as to obtain the associated information as shown in the following table.
Table three, data examples after correlation
| Field 1 | Field 2 | Field 3 | ||
| code1 | Data 1 | Clothing 1 | Color of | Red colour |
| code2 | Data of item 2 | Clothes 2 | Color of | Green, green |
| code1 | Data 3 rd | Clothing 1 | Size of the article | 38 |
| code2 | Data of item 4 | Clothes 2 | Size of the article | 40 |
So that the associated data can be synchronized to the target data table.
S203, determining a header of the target data table according to the multi-attribute field of the source data table, wherein the header comprises a plurality of attribute names;
in the embodiment of the invention, when the associated data is synchronized to the target data table, the code can be dynamically inserted into the target data table.
In a specific implementation, a plurality of preset attribute names may be written in the target data table, so that each attribute name corresponds to a field. The plurality of preset attribute names may be obtained by analyzing data recorded in the source data table. For example, for the source data table in the example above, the attribute names may include a clothing ID and three of color and size.
Thus, the header of the target data table may be expressed as:
table IV, header example of target data Table
| Data sequence number | Garment ID | Color of | Size of the article |
S204, respectively determining attribute values corresponding to all attribute names in the data with the same ID according to the dynamic codes;
then, according to the above dynamic encoding, attribute values corresponding to the respective attribute names in the data having the same ID can be determined, respectively.
For example, the attribute names corresponding to the clothing ID "clothing 1" include "color" and "size". Wherein, the attribute value corresponding to the color is red, and the attribute value corresponding to the size is 38.
S205, writing the attribute values into fields corresponding to the attribute names in the target data table one by one;
In the embodiment of the invention, corresponding data can be inserted into the target data table one by taking dynamic coding as a reference.
That is, it is first determined that the data record corresponding to code1 includes the 1 st and 3 rd pieces of data, and the clothing ID in the two pieces of records is written into the field corresponding to the clothing ID of the target data table, so as to obtain:
TABLE five
| Data sequence number | Garment ID | Color of | Size of the article |
| Data 1 | Clothing 1 |
Then, writing the corresponding color value and size value in the 1 st and 3 rd data records in the determined source data table into the corresponding fields in the target data table to obtain:
TABLE six
| Data sequence number | Garment ID | Color of | Size of the article |
| Data 1 | Clothing 1 | Red colour | 38 |
According to the method, the synchronization of all data records is completed, and the following steps are obtained:
Watch seven
S206, performing full synchronization on the source data table in preset idle time to obtain a check data table;
in the embodiment of the invention, the source data table can be used as a reference, and the synchronization of the full data is completed once, namely, all the data in the source data table are synchronized into the check data table.
Since the full-volume synchronization occupies more resources and takes longer time, the full-volume synchronization can be performed in idle time, such as early morning.
S207, comparing each field in the target data table with the corresponding field in the check data table;
The comparison of the target data with the data in the check data table may be performed field by field. For example, it may be first compared whether the data stored in the field Ba in the target data table B is compared with the data stored in the field Ca in the check data table C, and after the data in the field B is completely compared with the data stored in the field Cb in the target data table B, the data stored in the field Bb in the target data table B is compared with the data stored in the field Cb in the target data table C until the comparison of all the data is completed.
S208, if the data in any field in the target data table is different from the data in the corresponding field in the check data table, correcting the data in the target data table according to the data in the corresponding field in the check data table.
In the embodiment of the invention, if the data recorded by each field in the target data table is the same as the data recorded by the corresponding field in the check data table, the synchronized data can be considered to be complete.
If the data in any field in the target data table is different from the data in the corresponding field in the check data table, the error can be considered to occur in the synchronization process, and the data needs to be corrected.
Since the full-synchronization check data table is newly synchronized, the check data table should be in control when the data is corrected. Thus, different data in the target data table may be corrected to the same data as at the corresponding location in the check data table.
In the embodiment of the invention, when the integrity check is required to be carried out on the data, the source data table can be firstly subjected to full-quantity synchronization, then the integrity check is carried out by comparing the target data table after incremental synchronization with the check data table after full-quantity synchronization, and the two data tables are used for comparison and verification, so that the accuracy of the verification is improved.
It should be noted that, the sequence number of each step in the above embodiment does not mean the sequence of execution sequence, and the execution sequence of each process should be determined by its function and internal logic, and should not limit the implementation process of the embodiment of the present invention in any way.
Referring to fig. 3, a schematic diagram of an apparatus for data synchronization according to an embodiment of the present invention may specifically include the following modules:
The correspondence configuration module 301 is configured to configure a correspondence between a source data table and a target data table, where the correspondence includes a data table correspondence between the source data table itself and the target data table itself, and a field correspondence between each field in the source data table and each field in the target data table;
an entry number determining module 302, configured to determine a maximum value of data entry numbers in the target data table;
An execution statement generating module 303, configured to generate a data synchronous execution statement according to a maximum value of the number of the data entry in the target data table and a corresponding relationship between the data table and the field, where the data synchronous execution statement includes a query statement and an insert statement;
The to-be-synchronized data query module 304 is configured to execute the query statement to extract to-be-synchronized data from the source data table;
A data to be synchronized insertion module 305, configured to execute the insertion statement to insert the data to be synchronized into the target data table;
And the data verification module 306 is configured to perform integrity verification on the target data table.
In the embodiment of the present invention, the execution statement generation module 303 may specifically include the following sub-modules:
A field to be synchronized determining submodule, configured to determine a field to be synchronized in the source data table;
The query statement generation sub-module is used for generating a query statement aiming at the source data table according to the field to be synchronized in the source data table and the maximum value of the data item numbers in the target data table;
and the insertion statement generation sub-module is used for generating an insertion statement aiming at the target data table according to the data table corresponding relation and the field corresponding relation.
In the embodiment of the present invention, the fields in the source data table are multi-attribute fields, and the apparatus may further include the following modules:
the dynamic coding determining module is used for determining dynamic coding of each field in the source data table;
The data association module is used for associating the data under each field according to the dynamic codes so that the data with the same ID corresponds to the same dynamic code;
and the data synchronization module is used for synchronizing the associated data to the target data table.
In the embodiment of the present invention, the data synchronization module may specifically include the following sub-modules:
a target data table header determining submodule, configured to determine a header of the target data table according to the multi-attribute field of the source data table, where the header includes a plurality of attribute names;
The attribute value determining submodule is used for respectively determining attribute values corresponding to all attribute names in the data with the same ID according to the dynamic codes;
And the attribute value writing sub-module is used for writing the attribute values into fields corresponding to the attribute names in the target data table one by one.
In the embodiment of the present invention, the data verification module 306 may specifically include the following sub-modules:
the full synchronization sub-module is used for carrying out full synchronization on the source data table in preset idle time to obtain a check data table;
And the data comparison sub-module is used for comparing each field in the target data table with the corresponding field in the check data table.
In an embodiment of the present invention, the data verification module 306 may further include the following sub-modules:
And the data correction sub-module is used for correcting the data in the target data table according to the data in the corresponding field in the check data table if the data in any field in the target data table is different from the data in the corresponding field in the check data table.
For the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference should be made to the description of the method embodiments.
Referring to fig. 4, a schematic diagram of an apparatus of one embodiment of the present invention is shown. As shown in fig. 4, the apparatus 400 of the present embodiment includes: a processor 410, a memory 420, and a computer program 421 stored in the memory 420 and executable on the processor 410. The processor 410 performs the steps of the various embodiments of the method of data synchronization described above, such as steps S101 through S106 shown in fig. 1, when executing the computer program 421. Or the processor 410, when executing the computer program 421, performs the functions of the modules/units in the above-described device embodiments, for example, the functions of the modules 301 to 306 shown in fig. 3.
Illustratively, the computer program 421 may be partitioned into one or more modules/units that are stored in the memory 420 and executed by the processor 410 to accomplish the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing particular functions, which may be used to describe the execution of the computer program 421 in the device 400. For example, the computer program 421 may be divided into a correspondence configuration module, an entry number determination module, an execution statement generation module, a data query to be synchronized module, a data insertion to be synchronized module, and a data verification module, each of which specifically functions as follows:
The corresponding relation configuration module is used for configuring the corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises the data table corresponding relation between the source data table and the target data table and the field corresponding relation between each field in the source data table and each field in the target data table;
an entry number determining module, configured to determine a maximum value of the data entry numbers in the target data table;
The execution statement generation module is used for generating a data synchronous execution statement according to the maximum value of the number of the data item in the target data table and the corresponding relation between the data table and the field, wherein the data synchronous execution statement comprises a query statement and an insertion statement;
The data to be synchronized query module is used for executing the query statement to extract data to be synchronized from the source data table;
The data to be synchronized is inserted into the target data table by executing the insertion statement;
and the data verification module is used for carrying out integrity verification on the target data table.
The device 400 may be a computing device such as a desktop computer, a notebook computer, a palm top computer, and a cloud server. The device 400 may include, but is not limited to, a processor 410, a memory 420. It will be appreciated by those skilled in the art that fig. 4 is merely an example of device 400 and is not intended to limit device 400, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., device 400 may further include input-output devices, network access devices, buses, etc.
The Processor 410 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 420 may be an internal storage unit of the device 400, such as a hard disk or a memory of the device 400. The memory 420 may also be an external storage device of the device 400, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the device 400. Further, the memory 420 may also include both internal storage units and external storage devices of the device 400. The memory 420 is used to store the computer program 421 as well as other programs and data required by the device 400. The memory 420 may also be used to temporarily store data that has been output or is to be output.
It will be apparent to those skilled in the art that the foregoing functional unit and module divisions are merely illustrative for convenience and brevity of description. In practical applications, the above-mentioned functions may be distributed by different functional units and modules according to the needs, that is, the internal structure of the apparatus/terminal device is divided into different functional units or modules, so as to complete all or part of the functions described above. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present invention. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference may be made to related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other manners. For example, the apparatus/terminal device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. On the other hand, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by instructing the relevant hardware by a computer program, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of each of the method embodiments described above when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable storage medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable storage medium may include content that is subject to appropriate increases and decreases as required by jurisdictions and by jurisdictions in which such computer readable storage medium does not include electrical carrier signals and telecommunications signals.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limited thereto. Although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention.
Claims (7)
1. A method of data synchronization, comprising:
configuring a corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises a data table corresponding relation between the source data table and the target data table and a field corresponding relation between each field in the source data table and each field in the target data table;
Determining the maximum value of the number of the data item in the target data table;
determining a field to be synchronized in the source data table;
generating an SQL statement for searching data in the source data table, determining the maximum value of the data item number of the synchronized data in the target data table according to the corresponding relation between the source data table and the target data table, and generating a query statement for the source data table by adding the maximum value of the data item number into the SQL statement, wherein the query statement is used for searching data larger than the maximum value of the data item number;
Generating an insertion statement aiming at the target data table according to the data table corresponding relation and the field corresponding relation;
Executing the query statement to extract data to be synchronized from the source data table, wherein the data to be synchronized is data stored under a field to be synchronized in the source data table;
Executing the insertion statement to insert the data to be synchronized into the target data table;
Performing full synchronization on the source data table in preset idle time to obtain a check data table; and comparing each field in the target data table with the corresponding field in the check data table to carry out integrity check on the target data table.
2. The method of claim 1, wherein the fields in the source data table are multi-attribute fields, the method further comprising:
determining dynamic codes of all fields in the source data table;
according to the dynamic codes, associating the data in each field so that the data with the same ID corresponds to the same dynamic code;
and synchronizing the associated data to the target data table.
3. The method of claim 2, wherein the step of synchronizing the associated data to the target data table comprises:
Determining a header of the target data table according to the multi-attribute field of the source data table, wherein the header comprises a plurality of attribute names;
Respectively determining attribute values corresponding to each attribute name in the data with the same ID according to the dynamic codes;
And writing the attribute values into fields corresponding to the attribute names in the target data table one by one.
4. The method as recited in claim 1, further comprising:
And if the data in any field in the target data table is different from the data in the corresponding field in the check data table, correcting the data in the target data table according to the data in the corresponding field in the check data table.
5. An apparatus for data synchronization, comprising:
The corresponding relation configuration module is used for configuring the corresponding relation between a source data table and a target data table, wherein the corresponding relation comprises the data table corresponding relation between the source data table and the target data table and the field corresponding relation between each field in the source data table and each field in the target data table;
an entry number determining module, configured to determine a maximum value of the data entry numbers in the target data table;
The execution statement generation module is used for determining fields to be synchronized in the source data table; generating an SQL statement for searching data in the source data table, determining the maximum value of the data item number of the synchronized data in the target data table according to the corresponding relation between the source data table and the target data table, and generating a query statement for the source data table by adding the maximum value of the data item number into the SQL statement, wherein the query statement is used for searching data larger than the maximum value of the data item number; generating an insertion statement aiming at the target data table according to the data table corresponding relation and the field corresponding relation;
the data to be synchronized inquiry module is used for executing the inquiry statement to extract data to be synchronized from the source data table, wherein the data to be synchronized is data stored under a field to be synchronized in the source data table;
The data to be synchronized is inserted into the target data table by executing the insertion statement;
The data verification module is used for carrying out full synchronization on the source data table in preset idle time to obtain a verification data table; and comparing each field in the target data table with the corresponding field in the check data table to carry out integrity check on the target data table.
6. An apparatus comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, realizes the steps of the method of data synchronization according to any one of claims 1 to 4.
7. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of data synchronization according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910008352.4A CN109885614B (en) | 2019-01-04 | 2019-01-04 | Data synchronization method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910008352.4A CN109885614B (en) | 2019-01-04 | 2019-01-04 | Data synchronization method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109885614A CN109885614A (en) | 2019-06-14 |
| CN109885614B true CN109885614B (en) | 2024-06-28 |
Family
ID=66925527
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910008352.4A Active CN109885614B (en) | 2019-01-04 | 2019-01-04 | Data synchronization method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109885614B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112749227A (en) * | 2019-10-30 | 2021-05-04 | 北京国双科技有限公司 | Data synchronization method and device |
| CN113407551A (en) * | 2020-03-17 | 2021-09-17 | 北京同邦卓益科技有限公司 | Data consistency determining method, device, equipment and storage medium |
| CN111984684B (en) * | 2020-08-20 | 2021-07-23 | 北京捷报金峰数据技术有限公司 | Data processing method and device |
| CN112307124B (en) * | 2020-11-03 | 2024-09-27 | 深圳神拳互动科技有限公司 | Database synchronous verification method, device, equipment and storage medium |
| CN112395850B (en) * | 2020-11-05 | 2024-11-12 | 中国人寿保险股份有限公司 | A method, device and electronic device for checking data definition consistency |
| CN112181995B (en) * | 2020-11-06 | 2023-08-29 | 深圳数阔信息技术有限公司 | Data processing method, device, equipment and storage medium of data table |
| CN114595241A (en) * | 2022-02-11 | 2022-06-07 | 北京金山云网络技术有限公司 | Database data processing method and device, storage medium and electronic equipment |
| CN115687521A (en) * | 2022-10-27 | 2023-02-03 | 珠海格力电器股份有限公司 | Data synchronization method and device and computer equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103164422A (en) * | 2011-12-09 | 2013-06-19 | 百度在线网络技术（北京）有限公司 | Method, equipment and system for conducting data synchronism by adopting general data storage format |
| CN108121757A (en) * | 2017-11-10 | 2018-06-05 | 广州优视网络科技有限公司 | A kind of method of data synchronization, device, system, computing device and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108228814B (en) * | 2017-12-29 | 2022-02-15 | 泰康保险集团股份有限公司 | Data synchronization method and device |
| CN109062952B (en) * | 2018-06-22 | 2022-06-03 | 北京奇艺世纪科技有限公司 | Data query method and device and electronic equipment |
-
2019
- 2019-01-04 CN CN201910008352.4A patent/CN109885614B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103164422A (en) * | 2011-12-09 | 2013-06-19 | 百度在线网络技术（北京）有限公司 | Method, equipment and system for conducting data synchronism by adopting general data storage format |
| CN108121757A (en) * | 2017-11-10 | 2018-06-05 | 广州优视网络科技有限公司 | A kind of method of data synchronization, device, system, computing device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109885614A (en) | 2019-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109885614B (en) | Data synchronization method and device | |
| EP3678346B1 (en) | Blockchain smart contract verification method and apparatus, and storage medium | |
| CN107807982B (en) | Consistency checking method and device for heterogeneous database | |
| CN114281793B (en) | Data verification method, device and system | |
| CN109508355B (en) | A data extraction method, system and terminal device | |
| US8935575B2 (en) | Test data generation | |
| US9619492B2 (en) | Data migration | |
| US20140351239A1 (en) | Hardware acceleration for query operators | |
| CN108334609B (en) | Method, device, equipment and storage medium for realizing JSON format data access in Oracle | |
| CN112035471B (en) | Transaction processing method and computer equipment | |
| CN103593440A (en) | Method and device for reading and writing log file | |
| CN110569243B (en) | Data query method, data query plug-in and data query server | |
| CN111125229A (en) | Data blood margin generation method and device and electronic equipment | |
| CN110019111B (en) | Data processing method, data processing device, storage medium and processor | |
| CN109408035B (en) | Flow configuration method, storage medium and server of business system | |
| CN113407657A (en) | Data query method, device, equipment and storage medium based on single-level database | |
| CN115617773A (en) | Data migration method, device and system | |
| CN111221690B (en) | Model determination method and device for integrated circuit design and terminal | |
| CN107784073B (en) | Data query method for local cache, storage medium and server | |
| US8321429B2 (en) | Accelerating queries using secondary semantic column enumeration | |
| CN105550220A (en) | Fetching method and apparatus for heterogeneous system | |
| CN111459937B (en) | Data table association method, device, server and storage medium | |
| CN109324838B (en) | Execution method and execution device of single chip microcomputer program and terminal | |
| CN118035268A (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113076178B (en) | Message storage method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |