CN114116754B - Data updating method, device, equipment and storage medium of distributed database - Google Patents
Data updating method, device, equipment and storage medium of distributed database Download PDFInfo
- Publication number
- CN114116754B CN114116754B CN202111480374.4A CN202111480374A CN114116754B CN 114116754 B CN114116754 B CN 114116754B CN 202111480374 A CN202111480374 A CN 202111480374A CN 114116754 B CN114116754 B CN 114116754B
- Authority
- CN
- China
- Prior art keywords
- field
- target
- converted
- processed
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2379—Updates performed during online database operations; commit processing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data updating method of a distributed database, which comprises the steps of carrying out field extraction operation on each to-be-processed table to obtain each initial to-be-converted field, judging whether a target comparison relation corresponding to the initial to-be-converted field exists in a pre-established new and old field comparison table, if so, determining the comparison field corresponding to the initial to-be-converted field according to the target comparison relation, determining the comparison field as a target field, if not, determining the initial to-be-converted field as the target field, carrying out abstract calculation on each initial to-be-converted field and each target field by taking the to-be-processed table as a unit, determining the target to-be-converted field according to an abstract calculation result, carrying out deletion operation on each target to-be-converted field by calling and deleting scripts, and carrying out writing operation on the target field corresponding to each target to-be-converted field by calling and writing scripts. The invention improves the data updating efficiency and simplifies the data updating flow. The invention also discloses a device, equipment and a storage medium, which have corresponding technical effects.
Description
Technical Field
The present invention relates to the field of distributed storage technologies, and in particular, to a method, an apparatus, a device, and a computer readable storage medium for updating data of a distributed database.
Background
During the operation of the tax information system, the generated business data needs to be adjusted by part of field contents, such as tax agency codes, administrative division codes and the like, due to policy adjustment, agency adjustment, administrative division adjustment and the like.
In the prior art, an update script is written based on the field to be changed, and the data is updated through the operation and maintenance work execution script. Because the data operation is carried out around the database and table dividing keys, whether the database and table dividing keys are designated in the running process has great influence on the performance. If the field to be updated is a non-database-division table key field, the field is directly used as an updating condition to perform data updating operation, and full-table scanning can be generated in the data updating process, so that the operation and maintenance process is low in efficiency, the performance of the production database can be seriously influenced, the database can be dragged down, and the production business handling is influenced. In the case of the database and table dividing key, because of the change of the database and table dividing key, the deleting and writing script is needed to be manually generated, the existing data is deleted, the value of the database and table dividing field is updated, then the data is written, and the data is redistributed to a new database. Because the data volume is huge, the operation and maintenance process involves the deletion of the original data, and new data is organized based on the original data, the process is very complicated, errors are easy to occur, the workload of manual participation is large, and the processing efficiency is low.
In summary, how to effectively solve the problems of the whole table scanning that affects the production business handling, the operation and maintenance process is low in efficiency, the manual process of generating and deleting and writing the script is complex, error is prone to occur, the processing efficiency is low, and the like is a problem which needs to be solved by the people in the field.
Disclosure of Invention
The invention aims to provide a data updating method of a distributed database, which avoids full table scanning during updating of non-database-division and table-division key fields, improves data updating efficiency, simplifies data updating flow and improves data updating accuracy, and the invention also aims to provide a data updating device, equipment and a computer-readable storage medium of the distributed database.
In order to solve the technical problems, the invention provides the following technical scheme:
a method of data updating of a distributed database, comprising:
performing field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted;
judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table according to each initial field to be converted;
If yes, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, and determining the comparison field as a target field;
If not, determining the initial field to be converted as the target field;
Performing abstract calculation on each initial field to be converted by taking the table to be processed as a unit, performing abstract calculation on each target field, and determining the target field to be converted according to the abstract calculation result;
and calling the pre-generated deleting script to delete each target field to be converted, and calling the pre-generated writing script to write each target field corresponding to each target field to be converted.
In one embodiment of the present invention, with the to-be-processed table as a unit, performing summary calculation on each initial to-be-converted field, performing summary calculation on each target field, and determining a target to-be-converted field according to a summary calculation result, including:
merging the initial fields to be converted in each table to be processed according to a preset sequence, and performing abstract calculation on the merging result to obtain a first abstract field;
merging target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to the preset sequence, and performing summary calculation on the merging result to obtain a second summary field;
judging whether a corresponding relation between the first abstract field and the second abstract field exists in a pre-established conversion intermediate table according to each table to be processed;
If yes, determining a to-be-processed table with the corresponding relation between the first abstract field and the second abstract field in the conversion intermediate table as a target to-be-processed table, and determining each initial to-be-converted field in the target to-be-processed table as a target to-be-converted field.
In one embodiment of the present invention, performing a field extraction operation on to-be-processed data in each to-be-processed table includes:
analyzing the received field extraction request to obtain a field to be extracted;
Judging whether the field to be extracted is a database and table separating field;
if yes, determining each to-be-processed table according to the database and table dividing field, and performing field extraction operation on to-be-processed data in each to-be-processed table.
In a specific embodiment of the present invention, after deleting each of the target fields to be converted by invoking a pre-generated deletion script, the writing operation performed by invoking a pre-generated writing script on each of the target fields to be converted respectively corresponding to the target fields further includes:
putting the deleting script and the writing script into a script table to be executed in a production library supporting business handling;
deriving the deletion script and the write script from the script table to be executed;
and executing the deleting script to delete each target field to be converted in the production library by using a preset operation and maintenance tool, and executing the writing script to write the target fields corresponding to each target field to be converted into the production library.
In one embodiment of the present invention, when it is determined that the field to be extracted is not a separate-library and separate-table field, the method further includes:
judging whether the field to be extracted exists in a preset extraction table according to the range;
if yes, searching a target field extraction range corresponding to the field to be extracted from the preset according to-range extraction table;
And determining each to-be-processed table according to the target field extraction range, and performing field extraction operation on to-be-processed data in each to-be-processed table.
In a specific embodiment of the present invention, after deleting each of the target fields to be converted by invoking a pre-generated deletion script, the writing operation performed by invoking a pre-generated writing script on each of the target fields to be converted respectively corresponding to the target fields further includes:
acquiring target database and table dividing keys respectively corresponding to the tables to be processed of the target to-be-converted fields and target main keys respectively corresponding to the target to-be-converted fields;
searching each target field to be converted from a production library supporting business handling according to each target library division table key and each target main key;
and deleting each target field to be converted in the production library, and writing the target fields corresponding to each target field to be converted into the production library.
In a specific embodiment of the present invention, after performing a field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted, the method further includes:
And backing up each initial field to be converted into a preset backup table, and generating a backup batch number corresponding to the current backup.
A data updating apparatus of a distributed database, comprising:
The field extraction module is used for carrying out field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted;
the first judging module is used for judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table according to each initial field to be converted;
the first target field determining module is used for determining a comparison field corresponding to the initial field to be converted according to the target comparison relation when determining that the target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table, and determining the comparison field as a target field;
The second target field determining module is used for determining the initial field to be converted as the target field when determining that the target comparison relation corresponding to the initial field to be converted does not exist in the pre-established new and old field comparison table;
The target field to be converted determining module is used for carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining the target field to be converted according to the abstract calculation result;
And the field deleting and writing module is used for calling the pre-generated deleting script to delete each target field to be converted, and calling the pre-generated writing script to write the target fields corresponding to each target field to be converted respectively.
A data updating apparatus of a distributed database, comprising:
A memory for storing a computer program;
a processor for implementing the steps of the data updating method of the distributed database as described above when executing the computer program.
A computer readable storage medium having stored thereon a computer program which when executed by a processor implements the steps of a data updating method of a distributed database as described above.
The data updating method of the distributed database comprises the steps of carrying out field extraction operation on data to be processed in each table to be processed to obtain each initial field to be converted, judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table according to each initial field to be converted, if so, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, determining the comparison field as a target field, if not, determining the initial field to be converted as the target field, carrying out abstract calculation on each initial field to be converted, carrying out abstract calculation on each target field, determining the target field to be converted according to abstract calculation results, calling a pre-generated deletion script to carry out deletion operation on each target field to be converted, and calling a pre-generated writing script to carry out writing operation on each target field corresponding to each target field to be converted.
According to the technical scheme, the new and old field comparison tables storing the comparison relation between the initial fields to be converted and the comparison fields are preset, and after the initial fields to be converted are extracted, the target fields corresponding to the initial fields to be converted are determined according to the new and old field comparison tables. And carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining to obtain a final target field to be converted according to an abstract calculation result. And automatically invoking the pre-generated deleting script to delete each target field to be converted, and automatically invoking the pre-generated writing script to write the corresponding target field. The invention avoids full table scanning when updating the non-database-division table key field, improves the data updating efficiency and greatly reduces the influence on the production business handling. The deleting script and the writing script are automatically generated to delete and write the fields, so that the data updating flow is greatly simplified, and the accuracy of data updating is improved.
Correspondingly, the invention also provides a data updating device, equipment and a computer readable storage medium of the distributed database corresponding to the data updating method of the distributed database, which have the technical effects and are not repeated herein.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for updating data in a distributed database according to an embodiment of the present invention;
FIG. 2 is a flowchart of another implementation of a method for updating data in a distributed database according to an embodiment of the present invention;
FIG. 3 is a block diagram illustrating a data update apparatus for a distributed database according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating a distributed database data update apparatus according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a specific structure of a data updating device of a distributed database according to the present embodiment.
Detailed Description
In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 is a flowchart of a method for updating data of a distributed database according to an embodiment of the present invention, where the method may include the following steps:
s101, performing field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted.
Determining each to-be-processed table according to the data updating request, and when the data of the data tables in the distributed database is required to be updated, performing field extraction operation on the to-be-processed data in each to-be-processed table to obtain each initial to-be-converted field. If the field to be updated is a sub-library sub-table field, the accurate field extraction can be directly performed according to the sub-library sub-table field, and if the field to be updated is not the sub-library sub-table field, the field extraction can be performed according to a predetermined range.
S102, judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new field comparison table and a pre-established old field comparison table according to each initial field to be converted, if so, executing the step S103, and if not, executing the step S104.
And presetting a new field comparison table storing a comparison relation between the initial field to be converted and the comparison field. After field extraction operation is performed on the data to be processed in each to-be-processed table to obtain each initial to-be-converted field, judging whether a target comparison relation corresponding to the initial to-be-converted field exists in the pre-established new and old field comparison table according to each initial to-be-converted field, if so, executing step S103, and if not, executing step S104.
S103, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, and determining the comparison field as a target field.
When a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table aiming at a certain initial field to be converted, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, and determining the comparison field as a target field.
S104, determining the initial field to be converted as a target field.
And when determining that the target comparison relation corresponding to the initial field to be converted does not exist in the pre-established new and old field comparison table aiming at a certain initial field to be converted, determining the initial field to be converted as a target field.
And S105, carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining the target field to be converted according to the abstract calculation result.
After determining the target fields corresponding to the initial fields to be converted respectively, carrying out abstract calculation on the initial fields to be converted by taking the table to be processed as a unit, and carrying out abstract calculation on the target fields to obtain an abstract calculation result. The comprehensive abstract calculation result of each initial field to be converted and the comprehensive abstract calculation result of each target field in each table to be processed are obtained, and the target field to be converted which really needs to be updated is obtained by comparing the comprehensive abstract calculation result of each initial field to be converted with the comprehensive abstract calculation result of each target field.
S106, deleting the target fields to be converted by calling the pre-generated deleting script, and writing the target fields corresponding to the target fields to be converted by calling the pre-generated writing script.
And after determining the target fields to be converted according to the abstract calculation result, invoking the pre-generated deletion script to delete each target field to be converted, and invoking the pre-generated writing script to write the target fields corresponding to each target field to be converted. Therefore, the automatic and accurate update of the database data is realized, the workload of manual participation is reduced, the data update efficiency is improved, and the influence on the database performance is reduced.
According to the technical scheme, the new and old field comparison tables storing the comparison relation between the initial fields to be converted and the comparison fields are preset, and after the initial fields to be converted are extracted, the target fields corresponding to the initial fields to be converted are determined according to the new and old field comparison tables. And carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining to obtain a final target field to be converted according to an abstract calculation result. And automatically invoking the pre-generated deleting script to delete each target field to be converted, and automatically invoking the pre-generated writing script to write the corresponding target field. The invention avoids full table scanning when updating the non-database-division table key field, improves the data updating efficiency and greatly reduces the influence on the production business handling. The deleting script and the writing script are automatically generated to delete and write the fields, so that the data updating flow is greatly simplified, and the accuracy of data updating is improved.
It should be noted that, based on the above embodiments, the embodiments of the present invention further provide corresponding improvements. The following embodiments relate to the same steps as those in the above embodiments or the steps corresponding to the steps may be referred to each other, and the corresponding beneficial effects may also be referred to each other, which will not be described in detail in the following modified embodiments.
Referring to fig. 2, fig. 2 is a flowchart of another implementation of a method for updating data of a distributed database according to an embodiment of the present invention, where the method may include the following steps:
s201, analyzing the received field extraction request to obtain a field to be extracted.
When data updating is needed to be carried out on the data table in the distributed database, the request end generates a field extraction request and sends the field extraction request to the data updating management center. The data updating management center receives the field extraction request and analyzes the received field extraction request to obtain a field to be extracted.
S202, judging whether the field to be extracted is a database and table separating field, if so, executing the step S203, and if not, executing the step S204.
After the analysis is performed to obtain the field to be extracted, judging whether the field to be extracted is a sub-library sub-table field, if so, performing the step S203, otherwise, performing the step S204, and otherwise, performing the step S204.
S203, determining each to-be-processed table according to the database and table dividing fields, and performing field extraction operation on to-be-processed data in each to-be-processed table to obtain each initial to-be-converted field.
When the field to be extracted is determined to be a sub-library sub-table field, the accurate extraction of the field can be performed according to the sub-library sub-table field, each to-be-processed table is determined according to the sub-library sub-table field, and the field extraction operation is performed on to-be-processed data in each to-be-processed table to obtain each initial to-be-converted field, so that the accurate extraction of the field is realized, and the field extraction efficiency is improved.
S204, judging whether the field to be extracted exists in the preset extraction table according to the range, if so, executing the step S205, and if not, performing full table extraction to obtain each initial field to be converted.
And (3) establishing an extraction table stored with the corresponding relation between each field and each field extraction range in advance, when determining that the field to be extracted is not a sub-library sub-table field, indicating that accurate extraction of the field cannot be performed according to the sub-library sub-table field, judging whether the field to be extracted exists in the preset extraction table according to the range, if so, indicating that the field to be extracted can be performed according to the range, executing step S205, and if not, indicating that the field to be extracted cannot be performed according to the range, and performing full-table extraction to obtain each initial field to be converted.
S205, searching a target field extraction range corresponding to the field to be extracted from a preset extraction table according to the range.
When the fact that the field to be extracted exists in the preset range-based extraction table is determined, the field extraction can be performed according to the range, and the target field extraction range corresponding to the field to be extracted is searched from the preset range-based extraction table. The tax authority code extraction operation may be performed in the provincial range, for example, when the tax authority code needs to be updated.
S206, determining each to-be-processed table according to the target field extraction range, and performing field extraction operation on to-be-processed data in each to-be-processed table to obtain each initial to-be-converted field.
After a target field extraction range corresponding to the field to be extracted is searched from a preset extraction table according to the range, determining each table to be processed according to the target field extraction range, and performing field extraction operation on the data to be processed in each table to obtain each initial field to be converted. By extracting the fields according to the range, the whole table extraction is avoided, and the data extraction efficiency is improved.
S207, backing up each initial field to be converted into a preset backup table, and generating a backup batch number corresponding to the current backup.
After extracting each initial field to be converted, backing up each initial field to be converted into a preset backup table, and generating a backup batch number corresponding to the current backup. The backup operation is carried out on each initial field to be converted, so that follow-up data tracking is facilitated, and the follow-up quick search of the backup field is facilitated by setting the backup batch number.
S208, judging whether a target comparison relation corresponding to the initial field to be converted exists in the pre-established new and old field comparison table according to each initial field to be converted, if so, executing the step S209, and if not, executing the step S210.
S209, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, and determining the comparison field as a target field.
And S210, determining the initial field to be converted as a target field.
S211, merging the initial fields to be converted in each table to be processed according to a preset sequence, and performing abstract calculation on the merging result to obtain a first abstract field.
The merging sequence of the initial fields to be converted in each table to be processed is preset, the initial fields to be converted in each table to be processed are merged according to the preset sequence, and the merging result is subjected to summary calculation by using a message summary Algorithm (MESSAGE DIGEST Algorithm5, MD 5) to obtain a first summary field.
S212, merging target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to a preset sequence, and performing summary calculation on the merging result by using a message summary algorithm to obtain a second summary field.
And merging target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to a preset sequence, and performing abstract calculation on the merging result to obtain a second abstract field.
It should be noted that, the first and second summary fields are only used to distinguish the result of the summary calculation after merging the initial fields to be converted from the result of the summary calculation after merging the target fields, and there is no sequence or size division.
S213, judging whether the corresponding relation between the first abstract field and the second abstract field exists in the pre-established conversion intermediate table according to each to-be-processed table, if so, executing the step S214, and if not, outputting prompt information of the to-be-converted field meeting the preset requirement.
After merging the initial fields to be converted, merging the merging results to obtain a first summary field, merging the target fields, and merging the merging results to obtain a second summary field, judging whether the corresponding relation between the first summary field and the second summary field exists in the pre-established conversion intermediate table according to each to-be-processed table, if so, indicating that the initial fields to be converted in the processing table need to be subjected to field conversion operation, and executing step S214, if not, indicating that the initial fields to be converted in the processing table do not need to be subjected to field conversion operation, and outputting prompt information of the fields to be converted which do not meet the preset requirements.
In addition, when the data updating task is received each time, the conversion intermediate table is initialized according to the current updating task, so that the related data content of the previous data updating is cleared, and the interference to the current data updating is avoided.
S214, determining a to-be-processed table with the corresponding relation between the first abstract field and the second abstract field in the conversion intermediate table as a target to-be-processed table, and determining each initial to-be-converted field in the target to-be-processed table as a target to-be-converted field.
When the corresponding relation between the first abstract field and the second abstract field exists in a conversion intermediate table pre-established for a certain to-be-processed table, the field conversion operation is required to be carried out on the initial to-be-converted field in the processing table, the to-be-processed table with the corresponding relation between the first abstract field and the second abstract field in the conversion intermediate table is determined to be a target to-be-processed table, and all the initial to-be-converted fields in the target to-be-processed table are determined to be target to-be-converted fields.
S215, the pre-generated deleting script is called to delete the target fields to be converted, and the pre-generated writing script is called to write the target fields corresponding to the target fields to be converted respectively.
S216, when the field to be extracted is a database and table separating field, putting the deleting script and the writing script into a script table to be executed in a production database supporting business handling.
And pre-establishing a script table to be executed in a production library supporting business handling. When the field to be extracted is a database-division and table-division field, the deleting script and the writing script are put into a script table to be executed in a production database supporting business handling.
It should be noted that, if the target field to be converted is a non-separate-library-and-separate-table key, the value of the target field is directly used to replace the value of the target field to be converted when the data is updated, so that the data can be updated quickly, if the target field to be converted is a separate-library-and-separate-table key, the deletion operation of the target field to be converted needs to be performed by calling the pre-generated deletion script, and then the write operation of the target field corresponding to the target field to be converted is performed by calling the pre-generated write script.
S217, a deleting script and a writing script are exported from a script table to be executed.
After the deletion script and the write script are put into a script table to be executed in a production library supporting business transaction, the deletion script and the write script are exported from the script table to be executed.
S218, deleting the target fields to be converted in the production library by using a preset operation and maintenance tool, and writing the target fields corresponding to the target fields to be converted into the production library by executing a writing script.
And presetting an operation and maintenance tool for script execution in the production library, after a deletion script and a write-in script are exported from a script table to be executed, executing the deletion script by the preset operation and maintenance tool to delete each target field to be converted in the production library, and executing the write-in script to write the target fields corresponding to each target field to be converted into the production library.
S219, when the to-be-extracted fields exist in the preset range-based extraction table, the target database and table keys respectively corresponding to the to-be-processed tables of the to-be-converted fields and the target main keys respectively corresponding to the to-be-converted fields are obtained.
When the to-be-extracted fields exist in the preset range-based extraction table, the target database sub-table keys respectively corresponding to the to-be-processed tables to which the target to-be-converted fields belong and the target main keys respectively corresponding to the target to-be-converted fields are obtained.
S220, searching each target field to be converted from the production library supporting business handling according to each target library division table key and each target main key.
After the target database and table dividing keys respectively corresponding to the to-be-processed tables of the target to-be-converted fields and the target main keys respectively corresponding to the target to-be-converted fields are obtained, searching the target to-be-converted fields from a production database supporting business handling according to the target database and table dividing keys and the target main keys. By searching corresponding fields in the production library by utilizing the target library and table dividing keys and the target main keys, the field searching efficiency is greatly improved, and the updating speed can be ensured even under the condition of large data volume.
S221, deleting the target fields to be converted in the production library, and writing the target fields corresponding to the target fields to be converted into the production library.
After each target field to be converted is searched from the production library supporting business handling according to each target library table dividing key and each target main key, deleting each target field to be converted in the production library, and writing the target field corresponding to each target field to be converted into the production library. Therefore, the quick deletion of the target field to be converted in the production library and the quick writing of the target field are realized.
Corresponding to the above method embodiment, the present invention further provides a data updating device of the distributed database, where the data updating device of the distributed database described below and the data updating method of the distributed database described above can be referred to correspondingly.
Referring to fig. 3, fig. 3 is a block diagram illustrating a data updating apparatus for a distributed database according to an embodiment of the present invention, the apparatus may include:
The field extraction module 31 is configured to perform a field extraction operation on the data to be processed in each table to obtain each initial field to be converted;
a first judging module 32, configured to judge, for each field to be converted, whether a target comparison relationship corresponding to the field to be converted exists in the pre-established new and old field comparison table;
the first target field determining module 33 is configured to determine, when it is determined that a target comparison relationship corresponding to the initial field to be converted exists in the pre-established new and old field comparison table, a comparison field corresponding to the initial field to be converted according to the target comparison relationship, and determine the comparison field as a target field;
the second target field determining module 34 is configured to determine the initial field to be converted as a target field when it is determined that the target comparison relationship corresponding to the initial field to be converted does not exist in the pre-established new and old field comparison table;
the target to-be-converted field determining module 35 is configured to perform summary calculation on each initial to-be-converted field, perform summary calculation on each target field, and determine a target to-be-converted field according to a summary calculation result;
The field deleting and writing module 36 is configured to invoke a pre-generated deleting script to delete the to-be-converted fields of each target, and invoke a pre-generated writing script to write the to-be-converted fields of each target respectively corresponding to the to-be-converted fields.
According to the technical scheme, the new and old field comparison tables storing the comparison relation between the initial fields to be converted and the comparison fields are preset, and after the initial fields to be converted are extracted, the target fields corresponding to the initial fields to be converted are determined according to the new and old field comparison tables. And carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining to obtain a final target field to be converted according to an abstract calculation result. And automatically invoking the pre-generated deleting script to delete each target field to be converted, and automatically invoking the pre-generated writing script to write the corresponding target field. The invention avoids full table scanning when updating the non-database-division table key field, improves the data updating efficiency and greatly reduces the influence on the production business handling. The deleting script and the writing script are automatically generated to delete and write the fields, so that the data updating flow is greatly simplified, and the accuracy of data updating is improved.
In one embodiment of the present invention, the target to-be-converted field determining module 35 includes:
the first abstract field obtaining submodule is used for merging the initial to-be-converted fields in each to-be-processed table according to a preset sequence, and carrying out abstract calculation on the merging result to obtain a first abstract field;
The second abstract field obtaining submodule is used for merging target fields corresponding to each initial field to be converted in each table to be processed according to a preset sequence, and carrying out abstract calculation on a merging result to obtain a second abstract field;
The first judging sub-module is used for judging whether the corresponding relation between the first abstract field and the second abstract field exists in the pre-established conversion intermediate table according to each to-be-processed table;
The target to-be-converted field determining submodule is used for determining a to-be-processed table with the corresponding relation between the first abstract field and the second abstract field in the conversion intermediate table as a target to-be-processed table when the corresponding relation between the first abstract field and the second abstract field exists in the pre-established conversion intermediate table, and determining each initial to-be-converted field in the target to-be-processed table as a target to-be-converted field.
In one embodiment of the present invention, the field extraction module 31 includes:
The request analysis sub-module is used for analyzing the received field extraction request to obtain a field to be extracted;
the second judging submodule is used for judging whether the field to be extracted is a database and table separating field;
And the field extraction sub-module is used for determining each table to be processed according to the sub-table fields of the sub-library when the field to be extracted is determined to be the sub-table fields of the sub-library, and carrying out field extraction operation on the data to be processed in each table to be processed.
In one embodiment of the present invention, the apparatus may further include:
the script throwing module is used for throwing the deleting script and the writing script into a script table to be executed in a production library supporting business transaction after the deleting script generated in advance is called to delete each target field to be converted and the writing script generated in advance is called to write the target fields corresponding to each target field to be converted respectively;
The script exporting module is used for exporting a deleting script and a writing script from a script table to be executed;
the first production library field deleting and writing module is used for deleting each target field to be converted in the production library by using a preset operation and maintenance tool to execute a deleting script, and writing the target fields corresponding to each target field to be converted into the production library by executing a writing script.
In one embodiment of the present invention, the apparatus may further include:
The second judging module is used for judging whether the field to be extracted exists in the preset extraction table according to the range or not when the field to be extracted is determined not to be the field of the database and the table;
the extraction range searching module is used for searching a target field extraction range corresponding to the field to be extracted from the preset range extraction table when the field to be extracted exists in the preset range extraction table;
the field extraction module 31 is specifically a module that determines each table to be processed according to the target field extraction range, and performs a field extraction operation on the data to be processed in each table to be processed.
In one embodiment of the present invention, the apparatus may further include:
the main key acquisition module is used for acquiring target database and table keys respectively corresponding to the tables to be processed of the target to-be-converted fields and target main keys respectively corresponding to the target to-be-converted fields after the target to-be-converted fields are subjected to the deleting operation by the pre-generated deleting script and the writing operation by the pre-generated writing script;
The field to be converted searching module is used for searching each target field to be converted from a production library supporting business handling according to each target library splitting table key and each target main key;
and the second production library field deleting and writing module is used for deleting each target field to be converted in the production library and writing the target fields corresponding to each target field to be converted into the production library.
In one embodiment of the present invention, the apparatus may further include:
And the field backup and batch number generation module is used for carrying out field extraction operation on the data to be processed in each to-be-processed table to obtain each initial to-be-converted field, backing up each initial to-be-converted field into a preset backup table, and generating a backup batch number corresponding to the current backup.
Corresponding to the above method embodiment, referring to fig. 4, fig. 4 is a schematic diagram of a data updating apparatus for a distributed database provided by the present invention, where the apparatus may include:
a memory 332 for storing a computer program;
Processor 322, when executing the computer program, implements the steps of the data updating method of the distributed database of the method embodiment described above.
Specifically, referring to fig. 5, fig. 5 is a schematic diagram of a specific structure of a data updating apparatus of a distributed database according to the present embodiment, where the data updating apparatus of the distributed database may have relatively large differences according to different configurations or performances, and may include a processor (central processing units, CPU) 322 (e.g., one or more processors) and a memory 332, where the memory 332 stores one or more computer applications 342 or data 344. Wherein the memory 332 may be transient storage or persistent storage. The program stored in memory 332 may include one or more modules (not shown), each of which may include a series of instruction operations in the data processing apparatus. Still further, the processor 322 may be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the data update device 301 of the distributed database.
The data update device 301 of the distributed database may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input output interfaces 358, and/or one or more operating systems 341.
The steps in the data updating method of the distributed database described above may be implemented by the structure of the data updating apparatus of the distributed database.
Corresponding to the above method embodiments, the present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor, performs the steps of:
The method comprises the steps of carrying out field extraction operation on data to be processed in each to-be-processed table to obtain each initial to-be-converted field, judging whether a target comparison relation corresponding to the initial to-be-converted field exists in a pre-established new and old field comparison table for each initial to-be-converted field, if so, determining a comparison field corresponding to the initial to-be-converted field according to the target comparison relation and determining the comparison field as a target field, if not, determining the initial to-be-converted field as the target field, carrying out abstract calculation on each initial to-be-converted field by taking the to-be-processed table as a unit, carrying out abstract calculation on each target field, determining the target to-be-converted field according to an abstract calculation result, carrying out deletion operation on each target to-be-converted field by calling a pre-generated deletion script, and carrying out write operation on the target fields corresponding to each target to-be-converted field by calling the pre-generated write script.
The computer readable storage medium may include a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc. various media that can store program codes.
For the description of the computer-readable storage medium provided by the present invention, refer to the above method embodiments, and the disclosure is not repeated here.
In this specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, so that the same or similar parts between the embodiments are referred to each other. The apparatus, device and computer readable storage medium of the embodiments are described more simply because they correspond to the methods of the embodiments, and the description thereof will be given with reference to the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, but the description of the examples above is only for aiding in understanding the technical solution of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various modifications and adaptations of the invention can be made without departing from the principles of the invention and these modifications and adaptations are intended to be within the scope of the invention as defined in the following claims.
Claims (8)
1. A method for updating data in a distributed database, comprising:
performing field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted;
judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table according to each initial field to be converted;
If yes, determining a comparison field corresponding to the initial field to be converted according to the target comparison relation, and determining the comparison field as a target field;
If not, determining the initial field to be converted as the target field;
Performing abstract calculation on each initial field to be converted by taking the table to be processed as a unit, performing abstract calculation on each target field, and determining the target field to be converted according to the abstract calculation result;
The pre-generated deleting script is called to delete each target field to be converted, and the pre-generated writing script is called to write each target field corresponding to each target field to be converted;
Performing field extraction operation on the data to be processed in each table to be processed, including:
analyzing the received field extraction request to obtain a field to be extracted;
Judging whether the field to be extracted is a database and table separating field;
if yes, determining each to-be-processed table according to the database and table dividing field, and performing field extraction operation on to-be-processed data in each to-be-processed table;
When the field to be extracted is determined not to be a sub-library and sub-table field, the method further comprises:
judging whether the field to be extracted exists in a preset extraction table according to the range;
if yes, searching a target field extraction range corresponding to the field to be extracted from the preset according to-range extraction table;
determining each to-be-processed table according to the target field extraction range, and performing field extraction operation on to-be-processed data in each to-be-processed table;
Performing abstract calculation on each initial field to be converted, and performing abstract calculation on each target field, including:
And performing digest calculation on each initial field to be converted by using a message digest algorithm, and performing digest calculation on each target field by using the message digest algorithm.
2. The method for updating data in a distributed database according to claim 1, wherein performing digest calculation on each of the initial fields to be converted and digest calculation on each of the target fields in units of the tables to be processed, and determining the target fields to be converted based on the digest calculation result, comprises:
merging the initial fields to be converted in each table to be processed according to a preset sequence, and performing abstract calculation on the merging result to obtain a first abstract field;
merging target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to the preset sequence, and performing summary calculation on the merging result to obtain a second summary field;
judging whether a corresponding relation between the first abstract field and the second abstract field exists in a pre-established conversion intermediate table according to each table to be processed;
If yes, determining a to-be-processed table with the corresponding relation between the first abstract field and the second abstract field in the conversion intermediate table as a target to-be-processed table, and determining each initial to-be-converted field in the target to-be-processed table as a target to-be-converted field.
3. The method for updating data in a distributed database according to claim 1, wherein after the deleting operation is performed on each of the target fields to be converted by the invoking pre-generated deletion script, the writing operation is performed on each of the target fields to be converted by the invoking pre-generated writing script, which further includes:
putting the deleting script and the writing script into a script table to be executed in a production library supporting business handling;
deriving the deletion script and the write script from the script table to be executed;
and executing the deleting script to delete each target field to be converted in the production library by using a preset operation and maintenance tool, and executing the writing script to write the target fields corresponding to each target field to be converted into the production library.
4. The method for updating data in a distributed database according to claim 1, wherein after the deleting operation is performed on each of the target fields to be converted by the invoking pre-generated deletion script, the writing operation is performed on each of the target fields to be converted by the invoking pre-generated writing script, which further includes:
acquiring target database and table dividing keys respectively corresponding to the tables to be processed of the target to-be-converted fields and target main keys respectively corresponding to the target to-be-converted fields;
searching each target field to be converted from a production library supporting business handling according to each target library division table key and each target main key;
and deleting each target field to be converted in the production library, and writing the target fields corresponding to each target field to be converted into the production library.
5. The method for updating data in a distributed database according to claim 1, further comprising, after performing a field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted:
And backing up each initial field to be converted into a preset backup table, and generating a backup batch number corresponding to the current backup.
6. A data updating apparatus for a distributed database, comprising:
The field extraction module is used for carrying out field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted;
the first judging module is used for judging whether a target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table according to each initial field to be converted;
the first target field determining module is used for determining a comparison field corresponding to the initial field to be converted according to the target comparison relation when determining that the target comparison relation corresponding to the initial field to be converted exists in a pre-established new and old field comparison table, and determining the comparison field as a target field;
The second target field determining module is used for determining the initial field to be converted as the target field when determining that the target comparison relation corresponding to the initial field to be converted does not exist in the pre-established new and old field comparison table;
The target field to be converted determining module is used for carrying out abstract calculation on each initial field to be converted by taking the table to be processed as a unit, carrying out abstract calculation on each target field, and determining the target field to be converted according to the abstract calculation result;
the field deleting and writing module is used for calling a pre-generated deleting script to delete each target field to be converted, and calling a pre-generated writing script to write each target field corresponding to each target field to be converted;
The field extraction module includes:
The request analysis sub-module is used for analyzing the received field extraction request to obtain a field to be extracted;
the second judging submodule is used for judging whether the field to be extracted is a database and table separating field;
The field extraction sub-module is used for determining each table to be processed according to the sub-table fields of the sub-database when determining that the field to be extracted is the sub-table fields of the sub-database, and carrying out field extraction operation on the data to be processed in each table to be processed;
Further comprises:
The second judging module is used for judging whether the field to be extracted exists in the preset extraction table according to the range or not when the field to be extracted is determined not to be the field of the database and the table;
the extraction range searching module is used for searching a target field extraction range corresponding to the field to be extracted from the preset range extraction table when the field to be extracted exists in the preset range extraction table;
The field extraction module is specifically a module for determining each table to be processed according to the target field extraction range and performing field extraction operation on the data to be processed in each table to be processed;
The target field to be converted determining module is specifically a module for performing summary calculation on each initial field to be converted by using a message summary algorithm and performing summary calculation on each target field by using the message summary algorithm.
7. A data updating apparatus of a distributed database, comprising:
A memory for storing a computer program;
processor for implementing the steps of the data updating method of a distributed database according to any of claims 1 to 5 when executing said computer program.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the data updating method of a distributed database according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111480374.4A CN114116754B (en) | 2021-12-06 | 2021-12-06 | Data updating method, device, equipment and storage medium of distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111480374.4A CN114116754B (en) | 2021-12-06 | 2021-12-06 | Data updating method, device, equipment and storage medium of distributed database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114116754A CN114116754A (en) | 2022-03-01 |
| CN114116754B true CN114116754B (en) | 2025-05-13 |
Family
ID=80367060
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111480374.4A Active CN114116754B (en) | 2021-12-06 | 2021-12-06 | Data updating method, device, equipment and storage medium of distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114116754B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107958057A (en) * | 2017-11-29 | 2018-04-24 | 苏宁云商集团股份有限公司 | A kind of code generating method and device for being used for Data Migration in heterogeneous database |
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8165990B2 (en) * | 2009-06-15 | 2012-04-24 | Microsoft Corporation | Conversion of an application database to minimize time in single-user access mode |
| US10579644B2 (en) * | 2015-11-17 | 2020-03-03 | International Business Machines Corporation | Record property synchronization in a network computing system |
| CN110782313B (en) * | 2019-09-29 | 2022-12-23 | 苏宁云计算有限公司 | Database transaction processing method and device based on sub-database and sub-table |
| CN113065151A (en) * | 2020-08-27 | 2021-07-02 | 开鑫金服(南京)信息服务有限公司 | Relational database information security enhancement method, system, terminal and storage medium |
| CN113590714B (en) * | 2021-07-29 | 2023-09-08 | 东北石油大学 | A data synchronization method for relational databases |
-
2021
- 2021-12-06 CN CN202111480374.4A patent/CN114116754B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019211A (en) * | 2017-11-27 | 2019-07-16 | 北京京东尚科信息技术有限公司 | The methods, devices and systems of association index |
| CN107958057A (en) * | 2017-11-29 | 2018-04-24 | 苏宁云商集团股份有限公司 | A kind of code generating method and device for being used for Data Migration in heterogeneous database |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114116754A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112199366B (en) | Data table processing method, device and equipment | |
| CN103514223A (en) | Data synchronism method and system of database | |
| EP3327588A1 (en) | Value-id-based sorting in column-store databases | |
| CN109299101B (en) | Data retrieval method, device, server and storage medium | |
| CN111258990B (en) | Index database data migration method, device, equipment and storage medium | |
| EP2862101B1 (en) | Method and a consistency checker for finding data inconsistencies in a data repository | |
| CN114647651A (en) | Heterogeneous database synchronization method and system | |
| CN110990055B (en) | Pull Request function classification method based on program analysis | |
| CN118363870A (en) | Data automatic testing method and device and electronic equipment | |
| CN111858366A (en) | Test case generation method, device, equipment and storage medium | |
| CN115757479A (en) | Database query optimization method, machine-readable storage medium and computer device | |
| CN114116754B (en) | Data updating method, device, equipment and storage medium of distributed database | |
| US11886404B2 (en) | Automated database modeling | |
| CN111143582B (en) | Multimedia resource recommendation method and device for updating association words in double indexes in real time | |
| CN116303606A (en) | Method, storage medium and apparatus for generating database operation statement execution plan | |
| CN115309732A (en) | Method, device and equipment for shunting based on configuration request and storage medium | |
| CN114443728A (en) | Detection report searching method and device based on elastic search | |
| CN119938647B (en) | A database migration method, device, equipment, medium and product | |
| US10324919B2 (en) | Custom object paths for object storage management | |
| CN114253989B (en) | Data modification scheme detection method and device and electronic equipment | |
| JPH06180717A (en) | Database search method | |
| CN116187625B (en) | Space survey operation control resource scheduling plan comparison method and device based on bit operation | |
| CN115168399B (en) | Data processing method, device, equipment and storage medium based on graphical interface | |
| CN119671764B (en) | A reconciliation method, device, equipment, medium and product | |
| US12079179B2 (en) | Systems, methods, and media for accessing derivative properties from a post relational database utilizing a logical schema instruction that includes a base object identifier |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |