CN120215994A

CN120215994A - A database upgrade method

Info

Publication number: CN120215994A
Application number: CN202510271984.5A
Authority: CN
Inventors: 刘超
Original assignee: Beijing Qianze Technology Co ltd
Current assignee: Beijing Qianze Technology Co ltd
Priority date: 2025-03-07
Filing date: 2025-03-07
Publication date: 2025-06-27
Anticipated expiration: 2045-03-07
Also published as: CN120215994B

Abstract

The invention relates to the technical field of database analysis, in particular to a database upgrading method, which comprises the steps of establishing a dual-engine parallel environment, starting new and old version database engines, constructing a cross-version communication channel, responding to the established communication channel, carrying out parallel routing on a client request to the new and old engines through a transaction distributor, analyzing data change characteristics in real time and generating an increment synchronous instruction set based on a transaction operation log generated by the old engine, carrying out version coordinated transaction lock management when the new engine carries out data replay according to the generated increment synchronous instruction set, switching the transaction route to the new engine and closing the old engine writing channel after the data consistency verification of a preset period is completed, and realizing the aims of uninterrupted service and strong data consistency in the database version upgrading process through dual-engine parallel processing, increment data synchronization and cross-version lock coordination mechanisms.

Description

Database upgrading method

Technical Field

The invention relates to the technical field of database analysis, in particular to a database upgrading method.

Background

Conventional database version upgrade schemes generally face the dual challenges of poor service continuity and data consistency risks. In the prior art, the adoption of the shutdown migration mode can cause interruption of key business, and particularly in the scenes with high real-time requirements such as financial transactions, the internet of things and the like, the shutdown per hour can cause millions of economic losses. Although the online hot upgrading method reduces the downtime, the online hot upgrading method is limited by a serial processing mechanism of a single engine architecture, and still has the problems of long data migration window period, poor cross-version transaction compatibility and the like, and is extremely easy to cause data loss or inconsistent state. When large-scale data table structure changes or storage engine replacement are involved, the traditional scheme lacks an effective concurrency control means, and service response delay is often caused by too coarse lock granularity or inter-version deadlock is generated during fine-granularity lock management.

In addition, the static resource allocation strategy is difficult to cope with dynamic fluctuation of new and old engine loads in the upgrading process, and resource idling and bottleneck coexistence are easy to cause. In the prior art, data verification mostly adopts post-hoc full-quantity comparison, and incremental difference cannot be captured in real time, so that rollback decision is lagged. These drawbacks severely limit the usability and upgrade reliability of critical business systems.

Disclosure of Invention

The present invention is directed to a method for upgrading a database, so as to solve the problems set forth in the background art.

In order to achieve the above purpose, the invention provides a method for upgrading a database, comprising the following steps:

s1, establishing a dual-engine parallel environment, starting an engine of a new version database and an old version database, and constructing a cross-version communication channel;

S2, responding to the communication channel established in the step S1, and parallelly routing the client request to the new engine and the old engine for execution through the transaction distributor;

s3, analyzing data change characteristics in real time based on the transaction operation log generated by the old engine in the step S2 and generating an increment synchronization instruction set;

s4, implementing version coordinated transaction lock management when the new engine performs data replay according to the increment synchronous instruction set generated in the step S3;

S5, switching transaction routes to a new engine and closing an old engine writing channel after finishing data consistency verification of a preset period in the step S4, wherein the steps S1 to S5 realize the aims of uninterrupted service and strong data consistency in the process of upgrading the database version through double-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanisms.

As a further improvement of the present technical solution, the process of S1 specifically includes:

The method comprises the steps of deploying new and old version database examples in parallel in a computing cluster, distributing initial computing resources, setting a load sensing probe, and acquiring performance indexes of double engines according to the load sensing probe, wherein the performance indexes of the double engines comprise CPU utilization rate;

Establishing a bidirectional data pipeline comprising a multi-stage buffer queue, wherein the buffer queue comprises urgent, common and batch three-stage queues;

And triggering resource rebalancing according to the CPU utilization difference threshold value, and dynamically adjusting network bandwidth allocation based on the buffer filling rate.

As a further improvement of the present technical solution, the performing operation by the transaction distributor in S2 includes:

And performing double-engine synchronous execution and result comparison on the write operation, triggering the transaction rollback and recording the exception when the difference exceeds a preset threshold value.

As a further improvement of the present technical solution, the process of S3 specifically includes:

Analyzing the transaction operation log into an atomization data changing unit;

and merging according to the table dimension to generate a batch operation instruction set containing operation sequence constraint as an increment synchronization instruction set.

As a further improvement of the present technical solution, the transaction lock management in S4 includes dynamically selecting a lock policy according to an operation feature in the incremental synchronization instruction set, and specifically includes:

Dynamically selecting a table level lock and a row level lock according to the operation characteristics;

setting a lock timeout mechanism dynamically matched with the instruction execution time;

the dual engine lock state is synchronized through the cross-version communication channel.

As a further improvement of the present technical solution, the synchronous dual engine lock state includes:

when the double engines request the exclusive lock on the same resource, the old version lock is automatically released according to the preset priority;

allowing dual engines to simultaneously lock the same resource, supporting parallel reads.

As a further improvement of the present technical solution, the data consistency verification of the preset period includes:

Executing full data comparison and incremental change real-time comparison in a preset period;

Setting a fault tolerance threshold of consistency, and terminating the verification process when the fault tolerance threshold of consistency is exceeded.

As a further improvement of the present technical solution, the process of S5 specifically includes:

Suspending writing operation, switching transaction route and closing writing authority of old engine by stages;

Retaining an old engine reading function for monitoring verification after the switching;

and releasing the related network bandwidth of the old engine and performing new engine performance tuning.

As a further improvement of the present technical solution, the process of triggering resource rebalancing according to CPU utilization difference threshold specifically includes:

The method comprises the steps of acquiring CPU utilization rate indexes of the dual engines in real time, periodically calculating the difference value of the two indexes and comparing the difference value with a preset CPU utilization rate difference threshold, dynamically calculating a resource redistribution proportion by using a feedback control model based on the load weight coefficient of the current dual engines and the transaction priority when the difference value exceeds the CPU utilization rate difference threshold, and sending a quota adjustment instruction to a resource scheduler according to the distribution proportion, wherein the resource redistribution proportion is used for guaranteeing basic resource supply of the high-load engine preferentially.

As a further improvement of the present technical solution, the process of dynamically adjusting network bandwidth allocation based on the buffer filling rate specifically includes:

And monitoring the filling rate of the buffer area in real time, dynamically calculating the bandwidth allocation proportion according to the current load and the increasing trend when the filling rate threshold value of the buffer area is reached, optimizing the transmission parameters by utilizing a sliding window algorithm, and adjusting the bandwidth weight of each channel in real time by a flow controller to form a dynamic adaptation mechanism of the capacity of the buffer area and the network resources.

Compared with the prior art, the invention has the beneficial effects that:

The method for upgrading the database realizes zero business interruption in the process of upgrading the database through a dual-engine parallel processing architecture, combines a cross-version communication channel and an intelligent transaction distribution mechanism, ensures compatibility and integrity of a new version engine and a old version engine for cooperatively executing a transaction, generates a synchronous instruction set based on a transaction operation log analyzed in real time by an incremental data synchronization technology, cooperates with dynamic lock policy management, effectively reduces the risk of data conflict between versions, and ensures stable system throughput during upgrading by dynamically adapting network bandwidth and computing resources through a multistage buffer queue and a load-aware resource distribution mechanism.

In addition, through the consistency verification flow combining the full quantity and the increment, the data difference is rapidly identified in a preset period, the fault tolerance processing is triggered, the old engine reading function is reserved for abnormal rollback verification, and the whole scheme obviously improves the resource utilization efficiency and the system reliability in the upgrading process on the basis of ensuring the strong consistency of the data, and provides smooth upgrading guarantee for the key business system.

Drawings

FIG. 1 is a schematic diagram of the method steps of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Referring to fig. 1, the present invention provides a method for upgrading a database, comprising the following steps:

s1, a dual-engine parallel environment is established, an new version database engine and an old version database engine are started, and a cross-version communication channel is constructed, and the method specifically comprises the following steps:

the method comprises the steps of deploying new and old version database examples in parallel in a computing cluster, distributing initial computing resources (CPU core number, memory quota and storage volume) for the double engines, starting a load sensing probe, and collecting performance indexes of the double engines in real time, wherein the double engines represent the new and old version database engines;

establishing a bidirectional data pipeline, realizing cross-version data interaction between double engines, and setting a multi-level buffer queue, wherein the multi-level buffer queue comprises emergency, common and batch three-level queues;

Dynamically monitoring channel key indexes according to a bidirectional data pipeline, wherein the channel key indexes comprise buffer area filling rate, and the buffer area filling rate directly reflects quantization indexes of channel load states and is used for triggering dynamic bandwidth adjustment;

Periodically calculating the CPU utilization difference of the dual engines, triggering resource rebalancing when the difference exceeds a set threshold, and adjusting the resource quota of each engine according to a dynamic algorithm, wherein the resource rebalancing is triggered according to the CPU utilization difference threshold, and the method specifically comprises the following steps:

Periodically calculating the difference value of the two CPU utilization indexes by acquiring the double-engine CPU utilization index in real time and comparing the difference value with a preset CPU utilization difference value threshold; when the difference value exceeds the CPU utilization ratio difference value threshold value, dynamically calculating a resource redistribution proportion by using a feedback control model based on the current load weight coefficient and transaction processing priority of the dual engine, and using the feedback control model to preferentially ensure the basic resource supply of the high-load engine;

when the buffer filling rate of the bidirectional data pipeline is increased to a preset level, the network bandwidth allocation is automatically and proportionally increased, the self-adaptive supply capacity of channel resources is embodied, and the method specifically comprises the following steps:

By monitoring the change of the filling rate of the data buffer area in real time, when the filling rate is detected to exceed a preset buffer area filling rate threshold value, dynamically constructing a bandwidth allocation model based on the current network load state and the buffer area growth trend, and preferentially increasing the bandwidth quota for the high-filling rate buffer area channel;

And simultaneously, by combining historical transmission efficiency data, calculating the optimal bandwidth proportion by utilizing a sliding window algorithm, and adjusting the bandwidth allocation weight of each channel in real time by a flow controller to form a dynamic balance mechanism of buffer capacity and network resource supply, so that overflow or idle state of the buffer is effectively prevented, and the data transmission efficiency is continuously optimized.

The established communication channel realizes real-time data interaction and resource dynamic coordination between new and old engines through the bidirectional data pipeline and the multi-stage buffer queue, ensures that the double engines keep cooperative working capacity under load fluctuation, and provides infrastructure support for bidirectional flow control and data consistency guarantee in the subsequent stage.

S2, responding to the communication channel established in the step S1, and routing the client request to the new and old engines for execution through the transaction distributor, wherein the method specifically comprises the following steps:

Identifying the transaction type (query, update or structure change) of the client request, and extracting key characteristics (such as SQL operation mode and table object version dependence);

The real-time service capability index of the double engines is calculated by combining the CPU utilization rate and buffer filling rate of the double engines monitored in the S1 stage;

The write operation is simultaneously sent to the double engine for execution, the results returned by the double engines are compared field by field, when the time consumption difference of the double engine execution exceeds a preset threshold value, the transaction rollback is triggered and the exception is recorded, and the consistency of the operation is ensured through the result comparison;

In older versions of database engines, a log of transactional operations is recorded whenever a transactional operation (such as a query, update, or structure change) occurs. The log contains detailed information of the transaction, and the log collector such as Fluentd, logstash can read and forward the log file in real time by collecting the transaction operation log from the old version database engine in real time through the log collecting tool or agent program.

S3, analyzing the data change characteristics in real time and generating an increment synchronization instruction set based on the transaction operation log generated by the old engine in the step S2, wherein the method specifically comprises the following steps:

Continuously collecting a transaction operation log generated by an old version in the double engine, and analyzing log entries in the transaction operation log into an atomization data change unit;

Merging the analyzed atomization data change units into batch operation instructions according to the table dimension, and marking operation sequence constraint in the generated synchronous instruction set to serve as an increment synchronous instruction set.

S4, according to the increment synchronous instruction set generated in the step S3, implementing version coordination transaction lock management when the new engine performs data replay, and specifically comprising the following steps:

s4.1, dynamically selecting a lock strategy according to operation characteristics (such as batch writing and single-row updating) marked in the increment synchronous instruction set generated in the S3, wherein:

the table level lock is suitable for structure change or whole table data migration, and locks the whole table to ensure the atomicity of operation;

a row-level lock, which is used for locking only a target data row aiming at single-row data operation and maximizing concurrency performance;

And a lock timeout mechanism, namely setting the upper limit of the holding time of the lock, and dynamically matching with the estimated execution time of the instruction to prevent deadlock or long-term blockage.

S4.2, synchronizing the lock state (such as lock type, lock range and holding time) held by the old version in the double engine to the new version in real time through the cross-version communication channel established in the S1 stage, wherein:

When the dual engines request mutual exclusion locks on the same resource, the old version locks are automatically released according to preset priority (new version priority);

allowing the double engines to simultaneously add a shared lock to the same resource, and supporting parallel reading;

S4.3, defining a preset time period (such as 24 hours, 48 hours and the like), continuously verifying data consistency in the period, and performing full data comparison once when the verification period starts to ensure that all tables and data in the new and old engines are completely consistent;

in the verification period, continuously monitoring and recording incremental data changes of the new engine and the old engine, and performing real-time comparison, including transaction log analysis, data synchronization instruction set generation, real-time comparison and consistency judgment, wherein:

transaction log analysis, namely continuously collecting transaction operation logs generated by old versions in the double engines, and analyzing log entries into an atomization data change unit;

Merging the analyzed atomization data change units into batch operation instructions according to the table dimension, and marking operation sequence constraint in the generated synchronization instruction set;

real-time comparison, namely comparing the increment synchronous instruction executed by the new engine with the actual data of the old engine in real time to ensure that each change is correct and error-free;

Consistency determination defining a fault tolerance threshold for data consistency, such as a maximum allowable number or proportion of inconsistencies, continuously monitoring the number or proportion of inconsistencies during a verification period, and stopping the verification process immediately upon exceeding a set threshold.

S5, after finishing data consistency verification of a preset period in the step S4, switching a transaction route to a new engine and closing an old engine writing channel, namely executing write operation suspension, transaction route switching and old engine writing authority closing in stages, reserving an old engine reading function for monitoring verification after switching, releasing the related network bandwidth of the old engine and performing new engine performance tuning, wherein the specific steps comprise:

suspending write operation requests of all clients to ensure that no new data change occurs;

stopping sending the write operation to the operation modes executed by the two new and old engines simultaneously, wherein all read-write requests are temporarily suspended;

Modifying the configuration of the transaction distributor so that it only routes all subsequent transaction requests (including reads and writes) to the new version database engine;

after confirming that the new engine can independently process all types of operation requests, setting the new engine into an exclusive mode, wherein the new engine bears all workloads;

The write permission of the old engine is cut off to prevent any accidental data writing, but still retain its read function for a period of time to facilitate monitoring and verification.

The part of the bidirectional data pipeline relevant to the old engine is removed, network bandwidth and other resources which are not used any more are released, and simultaneously, the new engine is subjected to performance tuning.

The foregoing has shown and described the basic principles, principal features and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the above-described embodiments, and that the above-described embodiments and descriptions are only preferred embodiments of the present invention, and are not intended to limit the invention, and that various changes and modifications may be made therein without departing from the spirit and scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims

1. A method for upgrading a database, characterized in that the method steps are as follows:

S1. Establish a dual-engine parallel environment, start the old and new versions of the database engine, and build a cross-version communication channel;

S2, in response to the communication channel established in step S1, routing the client request to the new and old engines for execution in parallel through the transaction distributor;

S3. Based on the transaction operation log generated by the old engine in step S2, the data change characteristics are analyzed in real time and an incremental synchronization instruction set is generated;

S4. According to the incremental synchronization instruction set generated in step S3, version coordinated transaction lock management is implemented when the new engine executes data replay;

S5. After step S4 completes the data consistency verification for the preset period, switch the transaction routing to the new engine and close the old engine write channel; steps S1 to S5 use dual-engine parallel processing, incremental data synchronization and cross-version lock coordination mechanism to ensure that the service is not interrupted and the data maintains strong consistency during the database version upgrade process.

2. The database upgrade method according to claim 1, characterized in that the process of S1 specifically includes:

Deploy the old and new versions of database instances in parallel in the computing cluster, allocate initial computing resources and set up load-aware probes. Collect the performance indicators of the dual engines based on the load-aware probes, including CPU utilization.

Establishing a bidirectional data pipeline including a multi-level buffer queue, wherein the buffer queue includes three levels of queues: emergency, ordinary, and batch; dynamically monitoring channel key indicators according to the bidirectional data pipeline, wherein the channel key indicators include a buffer fill rate;

Resource rebalancing is triggered based on CPU utilization difference thresholds, and network bandwidth allocation is dynamically adjusted based on buffer fill rates.

3. The database upgrade method according to claim 1, wherein the transaction distributor in S2 performs the following operations:

The write operation is executed synchronously by two engines and the results are compared. When the difference exceeds the preset threshold, the transaction is rolled back and the exception is recorded.

4. The database upgrade method according to claim 1, characterized in that the process of S3 specifically includes:

Parse transaction operation logs into atomic data change units;

A batch operation instruction set containing operation order constraints is generated by merging table dimensions as an incremental synchronization instruction set.

5. The database upgrade method according to claim 1, characterized in that the transaction lock management in S4 includes dynamically selecting a lock strategy according to the operation characteristics in the incremental synchronization instruction set, specifically including:

Dynamically select table-level locks and row-level locks based on operation characteristics;

Set a lock timeout mechanism that dynamically matches the instruction execution time;

The dual-engine lock status is synchronized through the cross-version communication channel.

6. The database upgrade method according to claim 5, characterized in that the synchronous dual-engine lock state includes:

When two engines request a mutex lock for the same resource, the old version lock is automatically released according to the preset priority;

Allows dual engines to add shared locks to the same resource at the same time, supporting parallel reading.

7. The database upgrade method according to claim 1, wherein the data consistency verification of the preset period comprises:

Perform full data comparison and real-time comparison of incremental changes within a preset period;

Set a consistency tolerance threshold and terminate the verification process when the consistency tolerance threshold is exceeded.

8. The database upgrade method according to claim 1, characterized in that the process of S5 specifically includes:

Suspending write operations, switching transaction routing, and shutting down old engine write permissions are performed in stages;

Keep the old engine reading function for monitoring verification after switching;

Release the network bandwidth related to the old engine and perform performance tuning for the new engine.

9. The database upgrade method according to claim 2, wherein the process of triggering resource rebalancing according to the CPU utilization difference threshold specifically comprises:

By collecting dual-engine CPU utilization indicators in real time, the difference between the two is periodically calculated and compared with the preset CPU utilization difference threshold; when the difference exceeds the CPU utilization difference threshold, based on the current dual-engine load weight coefficient and transaction processing priority, the feedback control model is used to dynamically calculate the resource reallocation ratio to prioritize the basic resource supply of the high-load engine; and a quota adjustment instruction is sent to the resource scheduler based on the allocation ratio.

10. The database upgrade method according to claim 2, wherein the process of dynamically adjusting network bandwidth allocation based on buffer fill rate specifically comprises:

Monitor the buffer fill rate in real time. When the preset buffer fill rate threshold is reached, dynamically calculate the bandwidth allocation ratio based on the current load and growth trend, optimize the transmission parameters using the sliding window algorithm, and adjust the bandwidth weight of each channel in real time through the traffic controller to form a dynamic adaptation mechanism between buffer capacity and network resources.