CN121009094A - Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage media - Google Patents
Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage mediaInfo
- Publication number
- CN121009094A CN121009094A CN202510931243.5A CN202510931243A CN121009094A CN 121009094 A CN121009094 A CN 121009094A CN 202510931243 A CN202510931243 A CN 202510931243A CN 121009094 A CN121009094 A CN 121009094A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- storage
- information
- cutover
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本申请实施例公开了一种数据库详单数据获取方法、数据库详单数据获取装置、计算机设备和计算机可读存储介质。其中，方法包括如下步骤：解析获取到的控制指令确定第一数据库和割接需求；从第一数据库处获取第一数据库信息和第一数据库数据；判断第一数据库信息与割接需求是否匹配；若不匹配，则根据割接需求对第一数据库数据进行割接处理操作得到第二数据库数据；若匹配，则针对第一数据库数据进行预处理操作得到第二数据库数据；将第二数据库数据迁移至第二数据库；根据控制指令从第二数据库中获取目标数据；对目标数据执行传输操作得到传输数据，将传输数据返回至客户端。因此，本申请能够高效地从其他数据库中获取所需的详单数据。
This application discloses a method, apparatus, computer device, and computer-readable storage medium for obtaining database detail data. The method includes the following steps: parsing acquired control instructions to determine a first database and cutover requirements; obtaining first database information and data from the first database; determining whether the first database information matches the cutover requirements; if they do not match, performing cutover processing on the first database data according to the cutover requirements to obtain second database data; if they match, performing preprocessing on the first database data to obtain second database data; migrating the second database data to the second database; obtaining target data from the second database according to control instructions; performing a transmission operation on the target data to obtain transmission data; and returning the transmission data to the client. Therefore, this application can efficiently obtain the required detail data from other databases.
Description
Technical Field
The application belongs to the technical field of database management, and particularly relates to a database detail data acquisition method, a database detail data acquisition device, computer equipment and a computer readable storage medium.
Background
A large amount of data information such as various user data, business card data and the like at the server side of a plurality of software systems in the current society are stored in respective databases. In the operation process of the software system, the system is updated at a certain stage, and the software system can also cooperate and interact with other software. Therefore, in order to ensure the normal operation of the software system, an important requirement in switching, upgrading and merging the software system is to ensure that the history clients do not influence the normal use, and the history services can be accepted normally, so that the database data of the old system needs to be accurately converted into the new system database, and the new system database can be rapidly deployed and synchronized to the corresponding clients, thereby ensuring the normal use of the clients. However, the prior art can only meet one of the requirements, and fails to consider two points to improve the operation efficiency of the system. Therefore, how to obtain database data is a technical problem to be solved by those skilled in the art.
The foregoing description is provided for general background information and does not necessarily constitute prior art.
Disclosure of Invention
In view of the above, there is a need to provide a database detail data acquisition method, a database detail data acquisition apparatus, a computer device, and a computer-readable storage medium that can acquire desired detail data from other databases more efficiently.
The application solves the technical problems by adopting the following technical scheme:
The application provides a database detail data acquisition method which comprises the steps of analyzing an acquired control instruction to determine a first database and a cutting-over requirement, issuing the control instruction by a client, designating the acquisition of detail data in the database, acquiring first database information and first database data from the first database, judging whether the first database information is matched with the cutting-over requirement, if not, carrying out cutting-over processing operation on the first database data according to the cutting-over requirement to obtain second database data, if so, carrying out preprocessing operation on the first database data, marking the first database data after the preprocessing operation is carried out as second database data, migrating the second database data to the second database, wherein the second database is a database for storing data, acquiring target data from the second database according to the control instruction, carrying out transmission operation on the target data to obtain transmission data, and returning the transmission data to the client.
In an alternative embodiment of the present application, determining whether the first database information matches the cut-over requirement includes parsing the cut-over requirement to determine a target database type and a target database version number, obtaining a first database type and a first database version number in the first database information, determining that the first database type and the first database version number are not matched if the target database type is not equivalent to the first database type, determining whether the target database version number is compatible with the first database version number if the target database type is equivalent to the first database type, determining that the first database version number is not matched if the target database version number is not compatible with the first database version number, and determining that the first database version number is not matched if the target database version number is compatible with the first database version number.
In an alternative embodiment of the application, the cutting-over processing operation comprises the steps of obtaining a matching failure reason, analyzing first database data to determine data content of the first database data, dividing the data content into useful content and useless content, deleting the useless content to determine a key form in the useful content, reconstructing the key form according to cutting-over requirements to generate second database data, and obtaining a version conversion tool to process the first database data to obtain the second database data when the matching failure reason is the version number reason.
In an optional embodiment of the present application, the preprocessing operation includes obtaining a data format rule of the second database, where the data format rule is used to indicate a storage relationship of form data in the second database, adjusting data content of the first database data according to the data format rule until a preset conversion completion condition is met, where the preset conversion completion condition includes that the first database data has been completely converted into the second database data, and the first database data and the second database data have a corresponding relationship.
In an alternative embodiment of the application, the data migration of the second database to the second database comprises the steps of generating test data, simultaneously injecting the test data into a first database and a second database which run in parallel, judging whether the storage of the test data meets the consistency requirement, acquiring difference data if the storage does not meet the consistency requirement, adjusting a synchronization strategy according to the difference data, generating the test data again, injecting the test data into the first database and the second database until the consistency requirement is met, and completely migrating the data of the second database to the second database if the storage of the test data meets the consistency requirement.
In an alternative embodiment of the application, judging whether the storage of the test data meets the consistency requirement comprises acquiring storage information of the test data, wherein the storage information is used for indicating the state of the test data in a first database and a second database and comprises at least one item of storage position relation information, storage content information and storage operation information, analyzing the storage position relation information, judging whether the storage of the test data in the first database and the second database is carried out according to preset positions, judging whether the storage of the test data in the first database or the second database is carried out according to the preset positions, the storage position relation information is used for indicating the positions of the test data in a form of the first database or the second database and the relation with the rest of the test data, confirming that the consistency requirement is not met if the test data is not stored according to the preset positions, confirming that the consistency requirement is met if the test data is not stored according to the preset positions, and/or obtaining storage content information is used for indicating the stored data content of the first database or the second database, judging whether the storage content of the test data is identical or not stored in the first database, confirming that the test data is stored in the first database or the second database is stored in the different, confirming that the consistency requirement is met if the test data is not stored in the preset positions, confirming the consistency requirement is met, and/is confirmed when the test data is stored in the preset positions is met, and/is not stored in the preset ranges, and/is not stored in the first database, and the consistency requirement is confirmed when the consistency is met, and the consistency is not is judged, and the stored in a corresponding to be stored in the data is.
In an alternative embodiment of the application, the transmission operation comprises summarizing all transmission data, compressing the transmission data by a preset compression method to obtain compressed data, wherein the volume of the compressed data is smaller than that of the transmission data, generating a secret key according to the digital signature of the transmission data, encrypting the compressed data by the secret key, and marking the encrypted compressed data as the transmission data.
The application further provides a database detail data acquisition device, which comprises an analysis module, a cutting module, a processing module and an acquisition module, wherein the analysis module is used for analyzing acquired control instructions to determine a first database and cutting requirements, the control instructions are issued by a client and are used for appointing to acquire detail data in the database, the first database is a database for providing data, first database information and first database data are acquired from the first database, the cutting module is used for judging whether the first database information is matched with the cutting requirements, cutting processing operation is carried out on the first database data according to the cutting requirements to obtain second database data if the first database information is not matched with the cutting requirements, preprocessing operation is carried out on the first database data if the first database data is matched with the cutting requirements, the first database data after the preprocessing operation is executed is marked as second database data, the acquisition module is used for migrating the second database data to the second database, the second database is a database for storing data, target data are acquired from the second database according to the control instructions, transmission operation is carried out on the target data, and the target data are returned to the client.
The application also provides a computer device comprising a processor and a memory, the processor being arranged to execute a computer program stored in the memory to implement a method as described above.
The application also provides a computer readable storage medium storing a computer program which when executed by a processor implements a method as described above.
The embodiment of the application has the following beneficial effects:
The application aims at obtaining the detail list data across databases, and can convert the detail list data according to the difference between the two databases, so that the first database for providing the detail list data converts the first database data in the first database into the second database data meeting the requirements when the cutting requirements are not met, and stores the second database data in the second database. Therefore, when related detailed data are acquired later, the data can be directly acquired and returned from the second database, the intermediate additional operation flow is reduced, and the transmission efficiency is improved.
The foregoing description is only an overview of the present application, and is intended to be implemented in accordance with the teachings of the present application, as well as the preferred embodiments thereof, together with the following detailed description of the application, given by way of illustration only, together with the accompanying drawings. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application as claimed.
Drawings
In order to more clearly illustrate the embodiments of the application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Wherein:
FIG. 1 is a flowchart of a database detail data acquisition method according to an embodiment;
FIG. 2 is a flowchart for determining whether the first database information matches the cut-over requirement according to an embodiment;
FIG. 3 is a diagram showing the relationship between internal functional modules of a database detail data acquisition device according to one embodiment;
fig. 4 is a schematic block diagram of a computer device according to an embodiment.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In the server side of many software systems in the current society, for example, an internet settlement system is used for taking charge of service settlement (such as voice, short message and other services) among operators, service settlement needs to be checked firstly, and the service detail list (detail list) is checked all the time from each level of statistical list, so that the detail list of both sides needs to be imported. However, operators often adopt different databases for storing detailed data, so that in an application scenario similar to the above, the acquisition process needs to be performed across the databases, and the acquisition efficiency is extremely low.
Or many software systems in the current society can have the condition of updating the system, so the data cutting-over is frequently encountered in the working process. The broadcasting and television industry also needs to switch systems frequently under the conditions of provincial network unification, national network unification and the like as development needs, and finally one important requirement of the switching system is to ensure that historical clients do not influence normal use and historical services can be accepted normally, so that database data of an old system needs to be accurately converted into a newly obtained system database. I.e., the transfer of detailed data from one database to another.
In order to overcome the above technical problems, the present application provides a database detail data acquisition method, and for clarity of description of the method provided in this embodiment, please refer to fig. 1-2, including steps S110-S130.
Step S110, analyzing the acquired control instruction to determine a first database and a cutting-over requirement, wherein the control instruction is issued by a client and is used for appointing to acquire detailed list data in the database, the first database is a database for providing data, and the first database information and the first database data are acquired from the first database.
In an embodiment, the control instruction is issued by the client, and is used for designating to acquire the detailed list data in the database, which is usually set by a user according to the requirement, or may be automatically generated by the system, and the acquisition path is not particularly limited. It can be understood that the control instruction is an instruction for triggering to acquire the detail data, at least includes the specific content of the detail data to be acquired, and may also include a database storing the detail data, referred to as a first database, or may also be that after the control instruction is analyzed to determine the detail data to be acquired, the corresponding first database is determined by searching the detail data, which is not limited.
It will be appreciated that since the present application is directed to acquiring detail data across databases, the general idea of the present application is to splice the detail data from a first database to another database for storage by an additional splicing operation, for accepting another database storing data within the first database, or a new database, referred to herein as a second database. When the detailed data acquisition is carried out again later, the data do not need to be crossed with the database, and can be directly acquired from the second database, so that the flow of subsequent operation is reduced, and the acquisition efficiency is improved.
To achieve this, the first database information and the first database data need to be obtained from the first database. The first database information is used to indicate attribute information of the first database including, but not limited to, database type, database version, database data size, database data type, etc. The first database data is the data content actually stored in the first database. And the cutting requirement is used for indicating what conditions the first database is required to meet, namely the conditions belong to the form of meeting or being compatible with the second database. Specifically including at least one of the type of database to be converted into, and the specific version of the database. Based on both, the auxiliary completion database is cut. How to implement database cutoffs will be specifically developed later, where the table is first pressed.
Step S120, judging whether the first database information is matched with the cutting-over requirement, if not, cutting-over processing operation is carried out on the first database data according to the cutting-over requirement to obtain second database data, if so, preprocessing operation is carried out on the first database data, and the first database data after the preprocessing operation is marked as the second database data.
In one embodiment, determining whether the first database information matches the cut-over requirement includes parsing the cut-over requirement to determine a target database type and a target database version number, obtaining the first database type and the first database version number in the first database information, determining that the first database type and the first database version number are not matched if the target database type is not equivalent to the first database type, determining that the target database version number is compatible with the first database version number if the target database type is equivalent to the first database type, determining that the first database version number is not matched if the target database version number is not compatible with the first database version number, and determining that the first database version number is not matched if the target database version number is compatible with the first database version number.
In one embodiment, as described above, embodiments of the present application are directed to data acquisition across databases. Existing database types include, but are not limited to Oracle, sqlserver or mysql, etc., and version numbers between the first database and the second database even though the two databases are of the same type. The database data is cut from the first database to the second database, and there is a possibility that the database versions are not compatible with each other. Also as described above, the cutover requirement specifically records the conditions that need to be met to complete the cutover, and specifically includes at least one of the type of database that needs to be converted into, and the specific database version, that is, the target database type and the target database version number. While the first database information records a first database type and a first database version number. Thus, it may be determined what operation is to be performed on the first database to complete the database splice using whether the first database information and the splice requirement match. In this regard, reference may be made to a flowchart for determining whether the first database information matches the cutover requirement shown in fig. 2, which includes steps S210 to S250.
Step S210, analyzing the cutting requirements to determine the type and the version number of the target database, and acquiring the first database type and the first database version number in the first database information.
Step S220, judging whether the target database type is equal to the first database type.
In an embodiment, whether peer-to-peer, or, in particular, whether both are the same type of database. For example, if the target database type is sqlserver, the first database type is considered peer only if it is sqlserver, and the process goes to step S230, or else, if it is not peer, the process goes to step S240. Also, it is noted that neither the target database type nor the first database type may include one type, and that multiple types may be used. The condition of whether to peer can be arbitrarily set by the user. For example, the target database type is A, B, C, whether the peer is the first database type is A, B, C, or any one of the first database type and A, B, C, namely the peer is considered. The present application is not limited to the specific example, and may be arbitrarily set.
If not, step S230 is executed to perform the cutting processing operation on the first database data according to the cutting requirement to obtain the second database data.
If peer to peer, step S240 is performed to determine whether the target database version number is compatible with the first database version number.
In an embodiment, if not, step S230 is required to be performed, that is, a cutover processing operation is performed. The process execution steps will be described in detail later on for the cut-over processing operation, which is not developed here.
If step S240 determines that the version number of the target database is not compatible with the version number of the first database, step S230 is executed;
If the step S240 determines that the version number of the target database is compatible with the version number of the first database, a step S250 is executed to perform a preprocessing operation on the first database data, and the first database data after the preprocessing operation is marked as second database data.
In an embodiment, if it is determined in step S220 that the target database version number is peer-to-peer, it is necessary to further determine whether the target database version number is compatible with the first database version number. Typically, a new version is compatible with an old version, while an old version is not compatible with a new version. For incompatibility, the conversion is required, so that the step S230 is required, but for compatibility, the first database and the second database are basically the same or similar, only a small amount of processing is required, and even no processing is required, so that the database cut-over can be completed. For the case that the version types are equal and the version numbers are compatible, a preprocessing operation can be performed to complete the cutting process of the database. The preprocessing operation will be described in detail later, and is not developed here.
In one embodiment, the cutting-over processing operation comprises the steps of obtaining a matching failure reason, wherein the matching failure reason comprises a type reason and a version number reason, analyzing first database data to determine data content of the first database data when the matching failure reason is the type reason, dividing the data content into useful content and useless content, deleting the useless content to determine a key form in the useful content, reconstructing the key form according to cutting-over requirements to generate second database data, and obtaining a version conversion tool to process the first database data when the matching failure reason is the version number reason to obtain the second database data.
In one embodiment, as described above, the cutover processing is when the first database information is determined to be not equivalent or compatible in the foregoing determination. For this purpose, the reasons for the matching failure may be obtained before the processing steps are specifically performed, and the specific reasons may be determined, where the reasons for the matching failure include the type reason and the version number reason, which correspond to the cases of the incompatibility or the incompatibility described above, respectively. And executing different cutting-over processing operations aiming at different matching failure reasons.
When the failure cause is the type cause, analyzing the first database data to determine the data content of the first database data, dividing the data content into useful content and useless content, deleting the useless content, determining a key form in the useful content, and reconstructing the key form according to the cutting-over requirement to generate the second database data.
When the first database and the second database are not of the same type, for example the first database is an sqlserver or mysql database and the second database is an Oracle database. Then the first database data needs to be analyzed to complete the conversion. The first database data may be analyzed to determine the data content of the first database data, to differentiate the data content into useful content and non-useful content, to delete the non-useful content, and to determine key forms in the useful content, for example, by means of scripts converted to Oracle databases by the third party tool sqldeveloper. For a specific process, reference may be made to the following, connecting the first database, and then selectively migrating to the second database, and generating the ctl control file of the relevant table for representing the data content. And (3) deriving the dat file by using a bcp command, and filling the dat file into a table corresponding to the second database according to sqlldr and the format of the ctl control file.
And finding out the key table and the key information by analyzing the first database. Many resistances are encountered during the actual cut-over process, e.g. no database documents. The most perfect condition is that 1, a document description of a database table exists, 2, an old system interface can observe operation business, and 3, the table design of the other party is good, and the table comprises table names and field names of the table.
However, in many cases, these are not available, so that it is necessary to have the capability of analyzing the database, and the capability can be obtained through analysis by corresponding means. The first database is restored first, all tables of the first database are analyzed first, and statistics is performed from the data volume. By statistically analyzing the data amounts of each table, it is possible to analyze which tables in the first database may be useful tables, exclude empty tables, which tables are employee tables (data amounts are generally smaller), which tables are accepted service tables (data amounts are generally in the millions, tens of millions), and the like.
And then filtering the garbage table according to the analyzed rule, and picking out the useful table. Specifically, the data amount in the table can be queried, and the data amount is arranged in sequence from more to less according to the size of the data amount. The table whose table record is 0 is excluded, or only the table whose data amount is lower than the preset threshold value. The identification of whether a specific identifier 2 exists in the table name, for example, log, bak, his or the like, is not useful even though the amount of data is large, and can be eliminated. The remaining tables are analyzed one by one to find out some suspicious useful tables, such as client tables, product tables, commodity tables, employee tables, and the like, which are easy to judge like employees, departments, and the like, and generally include, but are not limited to, keywords such as operator, employee, dept, department, which can be preset, and are not limited to. The remaining tables of the screen are marked as useful tables.
In fact, many database designs are complicated in terms of security, and there are often cases where 1. Table names are irregular, 2. Field names are irregular, where the field names are first, it is difficult to guess what the specific meaning is, such as for administrative authorities, XZJG, 3. Storage is not canonical, for example, there is placing data on a spare field, such as file1 as a broadband account number, file2 as a customer name, and possibly even meaning inconsistent with the actual value, such as for access mode fields as address codes. Based on the above, some means are needed for analysis in order to find the correct table. Specifically, the following operation steps are available.
First, the full database user table contents are searched. For example, if a payment method is to be found, it is first determined that the payment method will generally be a small table, and then after defining the range, it may be determined which table and which field are to be located according to a full matching method, or according to a like fuzzy query method. The benefit of narrowing the query is that it does not wait too long for some tables to be data volumes of tens of millions, and the overall database search speed is slow.
Second, a certain field is included in the search column name. If the first database design is relatively more canonical, the corresponding field may be identified directly. Based on the customer data presented by the old system interface, such as customer code interface presentation 001, the code of this 001 can be used to search using the procedure stored in the first step above, or a specific data size range can be specified to search the relevant table, and finally locate which customer table is.
Finally, each table in the first database needs to be analyzed and excluded, so that the final useful content is reserved, and the key table in the useful content is determined. And reconstructing the key form according to the cutting-over requirement to generate the second database data. The data cutting-over method has the characteristics that a batch tool is required to be written, the method is characterized in that 1, sql is automatically executed, running time and running Log are recorded for error analysis, 2, time consumption of each sql is automatically recorded into a Log4JLogs table of hugetmep users, subsequent performance analysis is performed, 3, a database is automatically restored, and 4, the sql required to be executed is automatically combined and copied. Thereby completing the conversion of the first database data into the second database data.
And when the matching failure reason is the version number reason, the version conversion tool is acquired to process the first database data so as to obtain the second database data.
Specifically, taking the example that the first database and the second database are both Oracle databases, the version number of the second database is not compatible with the version number of the first database, e.g., the second database is older than the version of the first database. Thus, version conversion tools corresponding to the version numbers of the two databases can be obtained to realize conversion. For example, a 10g and 11g version distinction may be faced, and a generally higher version may be compatible with a lower version, but as the first database is a 11g library and the second database is a 10g library. The export needs to specify a version number of 10g, and if the data size is relatively large, the export is imported by the data pumps expdp and impdp as much as possible, so as to ensure the conversion from the first database data to the second database data.
In one embodiment, the preprocessing operation includes obtaining a data format rule of a second database, wherein the data format rule is used for indicating a storage relationship of form data in the second database, adjusting data content of first database data according to the data format rule until a preset conversion completion condition is met, and the preset conversion completion condition includes that the first database data is completely converted into the second database data and the first database data and the second database data have a corresponding relationship.
In one embodiment, the conversion may be accomplished by performing only a preprocessing operation for the case where the second database and the first database match. The data format rules of the second database are obtained, typically included in the cutover requirements. The data format rules are used to indicate the storage relationships of the form data within the second database, and may include information such as table structure definitions, field types, field lengths, data type conversion rules, and the like. The data format rules are the basis for implementing the data conversion from the first database to the second database.
The data content of the first database data is adjusted according to the data format rules, and the adjustment may include mapping of fields, conversion of data types, formatting of data values, and the like. For example, if the second database requires that a field be stored as an integer and the field be stored as a string type in the first database, the data type of the field needs to be converted. And the adjustment process gradually corrects the data in the first database according to the requirement of the data format rule until the preset conversion completion condition is met. The preset conversion completion condition comprises two sub-conditions that the first database data is completely converted into the second database data, namely, the data in the first database is adjusted according to the format requirement of the second database, and the converted data accords with the storage specification and the format of the second database. The first database data and the second database data have a corresponding relation, namely each converted data record can be in one-to-one correspondence with the corresponding record in the second database, and the integrity and consistency of the data are ensured. When the two conditions are met, the conversion process is completed, and an effective corresponding relation is established between the data in the first database and the data in the second database.
Step S130, migrating the second database data to a second database, wherein the second database is a database for storing data, acquiring target data from the second database according to a control instruction, performing transmission operation on the target data to obtain transmission data, and returning the transmission data to the client.
In one embodiment, transferring the second database data to the second database comprises generating test data, simultaneously injecting the test data into a first database and a second database which run in parallel, judging whether the storage of the test data meets the consistency requirement, acquiring difference data if the storage of the test data does not meet the consistency requirement, adjusting a synchronization strategy according to the difference data, generating the test data again, injecting the test data into the first database and the second database until the test data meets the consistency requirement, and completely transferring the second database data to the second database if the test data meets the consistency requirement.
In one embodiment, after the first database data is converted into the second database data, data migration may be performed to migrate the second database data to the second database. Before starting the migration process, a set of test data may be generated and injected simultaneously into the first database and the second database running in parallel. And judging whether the data in the first database and the second database meet the consistency requirement or not by detecting the storage condition of the test data. Whether the constant requirement is satisfied will be described in detail later.
If a consistency requirement is detected to be unsatisfied (e.g., the values of certain data fields do not match or are missing), the system automatically analyzes and obtains the difference data. The difference data may include, but is not limited to, different field values in the first database and the second database, missing or unsynchronized records, inconsistencies in data types, formats, and the like.
Based on these difference data, the synchronization strategy is automatically adjusted. For example, the order of data synchronization may be adjusted, field mapping rules may be changed, data type conversion rules may be adjusted, and so on. The adjustment of the synchronization policy may take various forms, such as modifying the batch size of the data migration, optimizing the concurrency during data transfer, and adjusting the time window during data synchronization.
After adjusting the synchronization strategy, test data is regenerated based on the new synchronization strategy and is again injected into the first database and the second database. And repeatedly carrying out consistency detection, and judging whether the adjusted synchronization strategy meets the consistency requirement. If the consistency detection result still does not meet the requirement, the difference data are continuously acquired, the synchronization strategy is adjusted, and the test data are regenerated for reinjection. This process is repeated until the data from both databases are completely consistent.
Once the system detects that the data consistency of the first database and the second database meets the preset requirement, the migration process enters a full-volume data migration stage. At this point, the system will begin to migrate all the data in the second database to the first database. In the data migration process, the system can ensure that each record of migration is matched with corresponding data in the target database, and consistency and integrity of the data after migration are ensured. After the full migration is completed, the system verifies whether the migrated data meets the consistency requirement or not, and confirms the correctness and the integrity of the data. In the migration process, if an abnormality occurs, the system rolls back the data migration and re-executes the steps until the data migration is successful.
In one embodiment, judging whether the storage of the test data meets the consistency requirement comprises acquiring storage information of the test data, wherein the storage information is used for indicating the state of the test data in a first database and a second database and comprises at least one of storage position relation information, storage content information and storage operation information, analyzing the storage position relation information, judging whether the storage of the test data in the first database and the second database is carried out according to preset positions, the storage position relation information is used for indicating the positions of the test data in a form of the first database or the second database and the relation with the rest of the test data, confirming that the consistency requirement is not met if the test data is not stored according to the preset positions, confirming that the consistency requirement is met if the test data is stored according to the preset positions, and/or judging that the storage content information is stored in the first database is not identical, judging whether the stored in the first database is identical to the first database or the second database is identical to the first database, confirming that the consistency requirement is met if the test data is not stored according to the preset positions, and/are not met, and/is judged that the storage content information is stored in the second database is stored in the first database is not identical to the first database is judged, judging that the stored content is stored in the storage data is identical to be identical to the first database, judging content is not stored in the first database is.
In one embodiment, the consistency requirement may be determined based on the stored information of the test data. The storage information is used for indicating states of the test data in the first database and the second database, and comprises at least one of storage position relation information, storage content information and storage operation information. And storing the position relation information, namely acquiring the storage position of the test data in the table through metadata of the database or an index system. The storage location relationship information may include names and field names of data tables, row locations of data records in the tables, relationships of data to other data, such as foreign key relationships, index relationships, and the like. Content information is stored indicating the data content stored in the database, including the field value of each record. This information may be obtained by querying a correlation table in a database. And storing operation information, namely indicating operation steps in the process of storing the data into the database. The method comprises operation time stamps of data insertion, updating and deleting, change histories of data in a database table, log information in the data operation process and the like.
Then, whether the consistency requirement is met or not can be judged according to the storage position relation information, the storage content information and the storage operation information. Analyzing the storage position relation information of the test data, and judging whether the storage positions of the test data in the two databases accord with preset positions or not. The predetermined location typically includes fields and record locations in a database table structure. The specific determination step is to compare whether the test data stored in the first database and the second database are in a predetermined table and field. For example, if test data should be stored in field B of table a, it is necessary to check whether the data in both databases is actually stored in this field. And comparing whether the position of the test data in the database accords with a preset relation. For example, if the order of data storage in the first database is inconsistent with the second database, it is determined that the data storage is inconsistent. If the test data is not stored according to the preset position, the data storage is determined to be inconsistent, the consistency requirement is not met, and the synchronization strategy adjustment is triggered. Otherwise, the next check is continued.
And judging whether the contents of the test data in the two databases are completely consistent according to the stored content information of the test data. The specific judging step is as follows, the content of the test data stored in the first database and the second database is obtained. The field values of the two are compared to determine if they are identical. For example, if a field in the first database stores "2024-12-23" as the date, and a field in the second database stores "12/23/2024", it is determined that the stored contents are inconsistent. And if the storage contents of the test data in the two databases are different, the test data are determined to not meet the consistency requirement, and difference data are returned to adjust the synchronization strategy. If the contents of the test data are consistent, the next check is continued.
Analyzing the storage operation information, and judging whether the difference exists in the operation steps executed in the data storage process. The specific judging steps are as follows, operation information of the test data stored in the two databases is obtained and analyzed, and operation steps of the data in the storage process are determined. Comparing the operation flow steps of the data stored in the first database with the operation flow steps of the data stored in the second database, such as the operation sequence of inserting, updating and deleting the data. Calculating the difference of the operation steps of the two, and if the difference of the operation steps exceeds a preset range (for example, the difference of the number of the operation steps is more than 5 steps), determining that the storage process is inconsistent. If the flow difference value is not in the preset range, judging that the consistency requirement is not met, returning difference data, and adjusting the synchronization strategy. If the difference value of the operation steps is within the preset range, the storage process is judged to be consistent, and the data migration or synchronization is continued.
In one embodiment, the transmission operation comprises the steps of summarizing all transmission data, compressing the transmission data through a preset compression method to obtain compressed data, wherein the volume of the compressed data is smaller than that of the transmission data, generating a secret key according to the digital signature of the transmission data, encrypting the compressed data through the secret key, and marking the encrypted compressed data as the transmission data.
In one embodiment, the second database stores all but not all of the detail data in the first database. For this purpose the target data may be retrieved from a second database according to a control command correspondence, the carrier being a csv or xlsx file. However, in the case of voice detail data, the data volume of one month is about eight million records, the data size of 1.5G is not compressed, and the data size after compression is about 200M. If conventionally conducted into a database, a record is inserted into the data table, requiring more than about 90 minutes. For this purpose, the target data needs to be compressed, so that a large number of detail importation is achieved. This can be achieved by a transmission operation.
Firstly, compressing the summarized data by using a preset compression method. The compression algorithm may be a standard compression algorithm, such as GZIP, ZIP, LZ77,77, etc., specifically chosen according to the data type and requirements. The goal of compression is to reduce the volume of data as much as possible, minimizing the bandwidth and storage space occupied during transmission. After compression, a key is generated from the digital signature of the transmitted data. Digital signatures are typically encrypted from a hash value and a private key of data, the purpose of which is to ensure that the data is not tampered with during transmission and to verify the source of the data. Assuming that the original data generates a hash value through an SHA-256 hash algorithm, then the hash value is encrypted by using a private key to obtain a digital signature. Based on this signature, the system will generate a corresponding symmetric or asymmetric encryption key for encrypting the compressed data. The generation process of the secret key ensures the security of encryption operation and corresponds to the digital signature one by one so as to ensure the integrity and the authenticity of data.
The compressed data is then encrypted using the generated key. The specific encryption method may be selected from symmetric encryption (e.g., AES) or asymmetric encryption (e.g., RSA). For large-scale data transmission, a symmetric encryption algorithm (e.g., AES) is typically used because it is more efficient than asymmetric encryption. Assuming that the AES algorithm is selected to be used, a 256-bit key is generated to encrypt the compressed data, ensuring that only the receiver holding the decryption key can recover the original data. Through the encryption operation, even if the data is intercepted in the data transmission process, the content of the data cannot be read by an unauthorized third party, and the confidentiality of the data is ensured.
Finally, the encrypted compressed data is marked as transmission data and ready for subsequent transmission. At this time, the data is already processed through compression and encryption, and the characteristics of efficient transmission and security protection are provided. In actual operation, the encrypted data may be stored in a transmission queue, or directly sent to a target server, a cloud storage system, or other network receiving end where the client is located. At this time, the encrypted compressed data marked as transmission data is subjected to subsequent processing as data to be transmitted. After receiving the data, the client decrypts the data by using the corresponding key, recovers the compressed data, and decompresses the data to restore the original data, thereby realizing the transmission of a large amount of detail data.
Therefore, when the detail list data is acquired across databases, the application can convert according to the difference between the two databases, so that the first database for providing the detail list data converts the first database data in the first database into the second database data meeting the requirements when the cutting requirements are not met, and stores the second database data in the second database. Therefore, when related detailed data are acquired later, the data can be directly acquired and returned from the second database, the intermediate additional operation flow is reduced, and the transmission efficiency is improved.
Fig. 3 shows a diagram of the internal functional blocks of the database detail data acquisition device in one embodiment. The database detail data acquisition device 300 comprises an analysis module 310, a cutover module 320 and an acquisition module 330. The parsing module 310 is configured to parse the obtained control instruction to determine a first database and a cutover requirement, where the control instruction is issued by the client and is used to specify and obtain detailed list data in the database, the first database is a database providing data, and obtain first database information and first database data from the first database. The cutting module 320 is configured to determine whether the first database information matches the cutting requirement, if not, perform cutting processing operation on the first database data according to the cutting requirement to obtain second database data, and if so, perform preprocessing operation on the first database data, and mark the first database data after the preprocessing operation is performed as the second database data. The obtaining module 330 is configured to migrate the second database data to the second database, where the second database is a database storing data, obtain the target data from the second database according to the control instruction, perform a transmission operation on the target data to obtain transmission data, and return the transmission data to the client.
FIG. 4 illustrates an internal block diagram of a computer device in one embodiment. The computer device may specifically be a terminal or a server. As shown in fig. 4, the computer device includes a processor, a memory, and a network interface connected by a system bus. The memory includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system, and may also store a computer program that, when executed by a processor, causes the processor to implement a database detail data acquisition method. The internal memory may also store a computer program which, when executed by the processor, causes the processor to perform the database detail data acquisition method. It will be appreciated by persons skilled in the art that the architecture shown in fig. 4 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, the present application also proposes a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of the method as described in any of the previous embodiments.
Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510931243.5A CN121009094A (en) | 2025-07-07 | 2025-07-07 | Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage media |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202510931243.5A CN121009094A (en) | 2025-07-07 | 2025-07-07 | Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage media |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN121009094A true CN121009094A (en) | 2025-11-25 |
Family
ID=97730831
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202510931243.5A Pending CN121009094A (en) | 2025-07-07 | 2025-07-07 | Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage media |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN121009094A (en) |
-
2025
- 2025-07-07 CN CN202510931243.5A patent/CN121009094A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7478113B1 (en) | Boundaries | |
| CN109325030B (en) | Message processing method, device, computer equipment and storage medium | |
| US20200412525A1 (en) | Blockchain filesystem | |
| US8443000B2 (en) | Storage of data with composite hashes in backup systems | |
| US7680998B1 (en) | Journaled data backup during server quiescence or unavailability | |
| EP2330519A1 (en) | Distributed file system and data block consistency managing method thereof | |
| US20040163029A1 (en) | Data recovery techniques in storage systems | |
| CN111177302A (en) | Business document processing method and device, computer equipment and storage medium | |
| CN111339551B (en) | Data verification method and related device and equipment | |
| KR20220086677A (en) | Systems and methods for blockchain-based backup and recovery | |
| US7949630B1 (en) | Storage of data addresses with hashes in backup systems | |
| CN112835918A (en) | An Implementation Method for Incremental Synchronization of MySQL Database | |
| CN109947730A (en) | Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing | |
| EP4292005B1 (en) | System and method for representing and verifying a data set using a tree-based data structure | |
| CN121009094A (en) | Database detailed data acquisition methods, devices, computer equipment, and computer-readable storage media | |
| CN118885318A (en) | A consistency detection method, system, device and medium | |
| US12493709B2 (en) | Managing a file transfer process for automatic database update | |
| CN120973599B (en) | IOS equipment data backup and recovery method and system | |
| CN101447886B (en) | Method for comparing mass data and device thereof | |
| CN119646041B (en) | Data encryption query methods, devices, equipment and storage media | |
| CN111079199B (en) | Enterprise credit data screenshot tamper-proofing method based on block chain technology | |
| CN118410115B (en) | Automatic double-activity disaster recovery method and device for ORACLE database and storage medium | |
| WO2024080883A1 (en) | Systems, methods and computer-readable media for migrating data | |
| CN119988497A (en) | Configurable database real-time synchronization method, device, equipment and storage medium | |
| CN119603141A (en) | Application configuration method, device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |