CN112860670B - Data migration method, data migration device, electronic equipment and computer storage medium - Google Patents
Data migration method, data migration device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN112860670B CN112860670B CN202110333945.5A CN202110333945A CN112860670B CN 112860670 B CN112860670 B CN 112860670B CN 202110333945 A CN202110333945 A CN 202110333945A CN 112860670 B CN112860670 B CN 112860670B
- Authority
- CN
- China
- Prior art keywords
- user
- access
- database
- data
- switching
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application relates to the technical field of databases and discloses a data migration method, a device, electronic equipment and a computer storage medium, wherein the data migration method comprises the following steps: determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all or part of the user data tables in the original database; and then, based on a preset route access strategy, a gray level switching table and a switching strategy, migrating the user data table to be migrated from the original database to a target database, and controlling the data access of the user data table to be migrated in the target database. The method of the embodiment of the application not only can finish stable data migration, but also does not stop external service during migration, reduces the influence of data migration in the database on the production of the client, avoids the condition of long-time transaction blocking caused by the shutdown of the application server, and realizes data migration without perception to the client.
Description
Technical Field
The embodiment of the application relates to the technical field of databases, in particular to a data migration method, a data migration device, electronic equipment and a computer storage medium.
Background
In recent years, channels on a bank retail line are rapidly developed, the user quantity is continuously and rapidly increased, the transaction number and the amount substitution rate of electronic banks are more than 94%, and the electronic channels become main channels of retail customers. Meanwhile, the contradiction between the rapid business development and the insufficient technical support is more and more acute, the problems of low throughput, high machine load, insufficient high concurrent business bearing capacity, serious single-point database dependence and the like of the traditional centralized database used by many banks are gradually exposed, and the performance bottleneck problem of the traditional centralized database on which the data storage is dependent becomes a key factor for restricting the sustainable development of an electronic channel.
In order to meet the requirements of massive user support, promote high concurrency and second killing service bearing capacity, quickly respond to market demands, search for distributed database migration is carried out by a plurality of banks successively, dirty data is prevented from occurring in order to keep consistency of data before and after data migration in the process of distributed database migration, and transaction of updating database data is required to be blocked in the data migration process because bank electronic channel user data has the characteristics of large data volume, high complexity, data sensitivity and high accuracy requirement, and meanwhile, in view of huge functions and various redundancy of service scenes of a service system, a shutdown strategy is generally adopted in the industry at present, namely, the transaction is prevented by stopping a server. The prior method is that before the distributed database is migrated, the application server of the system to be migrated is stopped, the application system stops providing service for the client, and the application server is restarted after the database is migrated.
However, the inventors of the present application found in the concrete implementation process that: the scheme of restarting the application server after the database is completely stopped until the database is migrated is completed, and the system has simple logic, low operation cost and difficult error, but the bank electronic channel business system belongs to a middle business system in a bank IT system, the availability of the system and the accuracy of user data are directly related to the use of the functions of the user electronic channel, and long-time stopping can lead the user to be unable to use any functions of the electronic channel in the time period, seriously influence the user experience and cause great damage to the credit and the public praise of the bank. In addition, the bank electronic channel system has large user data volume and high requirements on the safety and accuracy of the data, can not roll back after switching, has larger data recovery difficulty once production accidents occur after online, can possibly generate irreversible injury to users, and even can cause user complaints and has higher risk.
Disclosure of Invention
The aim of the embodiment of the application is to at least solve one of the technical defects, and the following technical scheme is specifically provided:
in one aspect, a data migration method is provided, including:
determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all or part of the user data tables in the original database;
Based on a preset route access strategy, a gray level switching table and a switching strategy, migrating the user data table to be migrated from an original database to a target database, and controlling the data access of the user data table to be migrated in the target database.
In one possible implementation manner, the gray level switching table and the switching strategy comprise a gray level switching table, a user switching table, a table switching information table and a transaction code and table correspondence table; wherein,
The gray level switch table is used for controlling the data access and data migration of the whole application;
the user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database;
the table switching information table stores ready states of the tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database;
The corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process.
In one possible implementation, the gray scale switch table includes a master switch, a change-over switch, and a new user access switch;
the master control switch is used for controlling the data access of the whole application and comprises an initial state, a gray state and an external release state;
the change-over switch is used for controlling a user or a data table to migrate from the original database to the target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over;
The new user access switch is used for controlling the new user to access the original database or the target database.
In one possible implementation, the data in the gray level switch table, the user switch table, the table switch information table and the transaction code and table correspondence table are all set into the remote dictionary service Redis by a timing task, so that the application reads the data from the Redis and writes the updated data into the Redis in real time when the data is updated.
In one possible implementation, when any one of the gray level switch table, the user switching table, the table switching information table and the transaction code and table correspondence table includes key user information, adding an attribute representing the user switching state to any one table for controlling database access of single user data;
wherein the key user information includes at least one of:
User number, mailbox, certificate number, security authentication mobile phone number, login mobile phone number, subscription account number, core client number, machine code, authentication system OpenID;
the user switching state includes non-switching, switching and switched.
In one possible implementation manner, before the user data table to be migrated is migrated from the original database to the target database based on the preset route access policy and the gray level switching table and the switching policy, the method further includes:
And dividing the user data table to be migrated into a user table and a parameter table according to whether the user data table to be migrated comprises key user information and an application scene of the user data table to be migrated.
In one possible implementation, the routing access policies include a user table access policy and a parameter table access policy;
the user table access strategy is used for controlling a user to access the original database or the target database, wherein if the master control switch is gray and the user identification is switched or the master control switch is issued outwards, the user is controlled to access the target database, otherwise, the user is controlled to access the original database;
The parameter table access strategy is used for controlling a user to access the original database or the target database, wherein if the parameter table state is that the external release or the gray period access mark is the target database, the user is controlled to access the target database, otherwise, the user is controlled to access the original database.
In one possible implementation, the routing access policy further includes an added user access policy;
the newly added user access strategy is used for determining whether the data of the newly added user is stored in the target database or the original database by configuring the state of a switch of the registered user in the database system parameter table.
In one aspect, a data migration apparatus is provided, including:
the determining module is used for determining a user data table to be migrated from the original database, wherein the user data table to be migrated is all user data tables or part of user data tables in the original database;
The processing module is used for migrating the user data table to be migrated from the original database to the target database based on a preset route access strategy, a preset gray level switching table and a preset switching strategy, and controlling the data access of the user data table to be migrated in the target database.
In one possible implementation manner, the gray level switching table and the switching strategy comprise a gray level switching table, a user switching table, a table switching information table and a transaction code and table correspondence table; wherein,
The gray level switch table is used for controlling the data access and data migration of the whole application;
the user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database;
the table switching information table stores ready states of the tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database;
The corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process.
In one possible implementation, the gray scale switch table includes a master switch, a change-over switch, and a new user access switch;
the master control switch is used for controlling the data access of the whole application and comprises an initial state, a gray state and an external release state;
the change-over switch is used for controlling a user or a data table to migrate from the original database to the target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over;
The new user access switch is used for controlling the new user to access the original database or the target database.
In one possible implementation, the data in the gray level switch table, the user switch table, the table switch information table and the transaction code and table correspondence table are all set into the remote dictionary service Redis by a timing task, so that the application reads the data from the Redis and writes the updated data into the Redis in real time when the data is updated.
In one possible implementation, when any one of the gray level switch table, the user switching table, the table switching information table and the transaction code and table correspondence table includes key user information, adding an attribute representing the user switching state to any one table for controlling database access of single user data;
wherein the key user information includes at least one of:
User number, mailbox, certificate number, security authentication mobile phone number, login mobile phone number, subscription account number, core client number, machine code, authentication system OpenID;
the user switching state includes non-switching, switching and switched.
In one possible implementation, the method further comprises a classification module; the classification module is used for:
And dividing the user data table to be migrated into a user table and a parameter table according to whether the user data table to be migrated comprises key user information and an application scene of the user data table to be migrated.
In one possible implementation, the routing access policies include a user table access policy and a parameter table access policy;
the user table access strategy is used for controlling a user to access the original database or the target database, wherein if the master control switch is gray and the user identification is switched or the master control switch is issued outwards, the user is controlled to access the target database, otherwise, the user is controlled to access the original database;
The parameter table access strategy is used for controlling a user to access the original database or the target database, wherein if the parameter table state is that the external release or the gray period access mark is the target database, the user is controlled to access the target database, otherwise, the user is controlled to access the original database.
In one possible implementation, the routing access policy further includes an added user access policy;
the newly added user access strategy is used for determining whether the data of the newly added user is stored in the target database or the original database by configuring the state of a switch of the registered user in the database system parameter table.
In one aspect, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data migration method described above when executing the program.
In one aspect, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the above-described data migration method.
According to the data migration method provided by the embodiment of the application, through the routing access strategy, the gray level switching table and the switching strategy, the data can be stably migrated through the routing access strategy control of the data table in the application layer during data migration, external service is not stopped during migration, the influence of the data migration in the database on the client in production is reduced, the situation of long-time transaction blocking caused by the shutdown of the application server is avoided, and the data migration which is not perceived by the client is realized; through the gray level release mechanism of the gray level switching table and the switching strategy, the full verification is carried out inside the gray level switching table and the switching strategy, and then the external popularization is carried out, so that the occurrence of production events is greatly reduced.
Additional aspects and advantages of embodiments of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of embodiments of the application will become apparent and may be better understood from the following description of embodiments with reference to the accompanying drawings, in which:
FIG. 1 is a flow chart of a data migration method according to an embodiment of the present application;
FIG. 2 is a schematic diagram illustrating an overall process of data migration according to an embodiment of the present application;
FIG. 3 is a flow chart of a user routing policy according to an embodiment of the present application;
FIG. 4 is a flowchart illustrating a user table database access according to an embodiment of the present application;
FIG. 5 is a flow chart of a parameter table write operation according to an embodiment of the present application;
FIG. 6 is a flowchart of an additional user access policy according to an embodiment of the present application;
FIG. 7 is a schematic diagram of a transaction layer control flow according to an embodiment of the present application;
FIG. 8 is an example process of a data migration process according to an embodiment of the present application;
FIG. 9 is a schematic diagram of a basic structure of a data migration apparatus according to an embodiment of the present application;
Fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the following detailed description of the embodiments of the present application will be given with reference to the accompanying drawings.
The following describes in detail the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above technical problems with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
One embodiment of the present application provides a data migration method performed by a computer device that is a server. The servers may be separate physical servers, clusters of physical servers, or virtual servers. As shown in fig. 1, the method includes:
Step S110, determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all or part of the user data tables in the original database; step S120, based on the preset route access strategy, gray level switching table and switching strategy, the user data table to be migrated is migrated from the original database to the target database, and the data access of the user data table to be migrated in the target database is controlled.
According to the data migration method provided by the embodiment of the application, through the routing access strategy, the gray level switching table and the switching strategy, the data can be stably migrated through the routing access strategy control of the data table in the application layer during data migration, external service is not stopped during migration, the influence of the data migration in the database on the client in production is reduced, the situation of long-time transaction blocking caused by the shutdown of the application server is avoided, and the data migration which is not perceived by the client is realized; through the gray level release mechanism of the gray level switching table and the switching strategy, the full verification is carried out inside the gray level switching table and the switching strategy, and then the external popularization is carried out, so that the occurrence of production events is greatly reduced.
According to the embodiment of the application, from the perspective of customer experience priority, the influence of database migration on customers in production is reduced, the situation of long-time transaction blocking caused by shutdown of an application server is avoided, meanwhile, from the perspective of risk management, if a problem occurs in a switching process, a rollback strategy capable of responding in time is necessary, the production problem is ensured to be solved in the first time, and an application layer flow control strategy for carrying out distributed database migration without shutdown is finally provided aiming at the complexity and safety of database migration of a bank application system. The following specifically describes an embodiment of the present application by taking data migration of a database in a banking system as an example. It should be noted that, in this example, the original database is the distributed database DB2, and the target database is the MySQL database.
The DB2 is a relational database management system, and the main running environments of the relational database management system are UNIX, linux and Windows servers. DB2 is mainly applied to a large-scale application system, has better scalability, can support from a mainframe to a single-user environment, and is applied to all common server operating system platforms. DB2 provides a high level of data utilization, integrity, security, recoverability, and small to large scale application execution capabilities with platform-independent basic functions and SQL commands. The DB2 adopts a data grading technology, so that mainframe data can be conveniently downloaded to a LAN database server, a client/server user and an application program based on the LAN can access the mainframe data, and the database localization and remote connection are transparent. DB2 is known to possess a very complete query optimizer, whose external connections improve query performance and support multitasking parallel queries. DB2 has good network support capability, each subsystem can be connected with hundreds of thousands of distributed users, thousands of active threads can be activated at the same time, and the method is particularly suitable for large-scale distributed application systems.
The MySQL database service is a completely hosted database service, is a relational database management system of open source codes, and uses the most commonly used database management language, namely the Structured Query Language (SQL), for database management.
In the data migration process, in order to ensure stable migration of data in production and prevent production events, the distributed database migration adopts a gray level release strategy during production, the production is divided into three stages of before switching, gray level release and formal external release, the gray level release stage is carried out, users to be switched (the users in the part are called white list users) are verified, and after verification is passed, flow switching is gradually carried out, so that formal external release is carried out.
Fig. 2 shows the overall process of data migration based on the gray level distribution strategy, in fig. 2 ① is the pre-commissioning phase, where the distributed database only accesses DB2, ②-⑤ is the gray level distribution phase, where the transformation commissioning is applied, where the accessing MySQL and accessing DB2 coexist, and the data migration is started gradually. The method comprises the steps of performing full migration on a user data table of tens of millions in advance, and performing daily incremental migration; and carrying out total migration daily on the service parameter type table with smaller data magnitude. The user migration adopts a strategy of firstly small-scale user migration, then small-batch migration, then large-batch migration and finally full-scale user migration to the MySQL database. After migration is completed, the database completely accesses the MySQL database, and after waiting for smooth operation, the original DB2 accesses the branch to be disconnected. If serious production problems occur, the database access rollback of all data tables, a single data table and a single user can be respectively realized by switching the master control switch, the table level switch and the user level switch.
When the distributed database is migrated and put into production, the application adopts a gray level release scheme, and the formal release is switched after verification is passed. And respectively configuring a gray level switch table, a user switch table, a table switch information table and a transaction code and table corresponding relation table during gray level production, namely adopting a gray level switch table and a switch strategy, wherein the gray level switch table and the switch strategy comprise the gray level switch table, the user switch table, the table switch information table, the transaction code and the table corresponding relation table.
In one possible implementation, the gray-scale switch table is used to control data access and data migration of the entire application; the user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database; the table switching information table stores ready states of the tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database; the corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process. The gray level switch table comprises a master control switch, a change-over switch and a newly added user access switch; the master control switch is used for controlling the data access of the whole application and comprises an initial state, a gray state and an external release state; the change-over switch is used for controlling a user or a data table to migrate from the original database to the target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over; the new user access switch is used for controlling the new user to access the original database or the target database.
In practical applications, (1) the data of the gray-scale switching table includes: a. the master control switch comprises an initial state, a gray level state and an external release state and controls the data access of the whole application; b. the change-over switch comprises three states of not starting, switching and switching completion, and controls a user or a table to migrate from DB2 to MySQL; c. the newly added user access switch comprises two states of newly added user access DB2 or MySQL. Greyscale control adds a new user to DB2 or MySQL. (2) a user switching table: storing the access DB2 and MySQL identifications of the users to be switched, and controlling the access of the database of the user level. (3) a table switching information table: storing ready identification (not ready or ready) of a to-be-switched table, a table type (a user table or a parameter table), a table access route (DB 2 or MySQL) for controlling database access of a data table level, and realizing access control of the whole table by switching the state of the switch, wherein when not ready, the DB2 is accessed, and when ready, the MySQL is accessed; (4) transaction code and table correspondence table: and in the switching process of the storage list or the user switching process, the transaction list is blocked.
The four tables are only relevant to gray level verification, are only configured in the DB2 database, are not migrated to the MySQL database later, and are directly connected after the application switching formally releases the data externally and operates stably.
In one possible implementation manner, in order to ensure that gray level switching or white list information change takes effect in real time and normal operation of application, and reduce database access frequency during application operation, data of the four tables is written into Redis by a timing task, read Redis during data reading is performed during application, real-time Redis writing is performed during data updating, and such updating is not frequent. The data in the gray level switch table, the user switching table, the table switching information table and the corresponding relation table of the transaction code and the table are all fixed in the remote dictionary service Redis by timing tasks, so that the application can read the data from the Redis and write the updated data into the Redis in real time when the data is updated. The Redis (Remote Dictionary Server) is an open source API (application program interface) which is written by using ANSI C language, supports network, can be based on a memory and can be persistent log type, key-Value database and provides multiple languages; redis is a high-performance key-value database.
In one possible implementation, when any one of the gray level switch table, the user switching table, the table switching information table and the transaction code and table correspondence table includes key user information, adding an attribute representing the user switching state to any one table for controlling database access of single user data; wherein the key user information includes at least one of: user number, mailbox, certificate number, security authentication mobile phone number, login mobile phone number, subscription account number, core client number, machine code, authentication system OpenID; the user switching state includes non-switching, switching and switched.
In other words, the attribute of user switching status is newly added to the data table related to key user information (i.e. 11 element information, 11 element information includes user information and user information types, wherein the user information includes user number, login phone number, card number, etc., and the user information types include user number, core client number, certificate number, login phone number, mailbox, security authentication phone number, DN, DN_CN, CN number, machine code, openID, subscription account, etc.), and the attribute is used for controlling database access of single user data. The user switching state includes three states: an unswitched state (which may be represented by the numeral 0), an in-switch state (which may be represented by the numeral 1), and a switched state (which may be represented by the numeral 2). During gray scale, the user switch state is used together with the switch to determine whether the user performs transaction blocking and whether the database accesses DB2 or MySQL during transaction. The OpenID is a digital identification framework centered on the user, and has openness and dispersibility.
Regarding the routing access policy in the embodiment of the application, the high-frequency data table to be migrated is divided into two main types of user tables and parameter tables according to whether key user information and application scenes are contained or not in the migration, three routing access methods are provided in a public component according to different operations of adding, deleting and modifying the two types of data tables, and the three routing access methods are respectively the user table access policy, the parameter table access policy and the newly added user access policy. According to different application scenes of adding, deleting and checking different data tables, different routing methods are called before a DAO layer database access statement method body to carry out adaptive transformation so as to realize that a user accesses DB2 or MySQL through routing control, influence on an upper-layer service processing component is reduced, and in addition, the service component can be transformed and packaged aiming at transactions, complex database access and the like.
1. User table access policy
Adding a user access routing method in a public component: whiteChangeRoute (table name, white list user information, user information type), wherein the white list user (i.e. user table) information includes user number, login phone number, card number, etc. The user information types include user number, core client number, certificate number, login phone number, mailbox, security authentication phone number, DN, DN_CN, CN number, machine code, openID and subscription account number. The method controls the user to access DB2 or MySQL, when the master control switch is gray, the user identification is switched, or the master control switch is issued outwards, the MySQL is accessed, and other conditions access DB2. The flow of the routing policy of the white list user is shown in fig. 3; the flow of access to the user table database is shown in fig. 4.
It should be noted that, the user table performs unilateral read-write operation during the gray scale period, before the gray scale starts, the full-scale migration is performed on the stored user table, and then during the gray scale period, the DB2 incremental data is migrated into MySQL every day.
2. Parameter table access policy
And adding an access routing method parTable (table name) of a parameter table in the public component, wherein when the parameter table state is that the externally published or gray period access identifier is MySQL, the MySQL is accessed, and other conditions access the DB2.
For the writing operation of the parameter table, the parameter table carries out bilateral writing operation during gray scale, namely, DB2 and MySQL are synchronously updated, DB2 is accessed during rollback, mySQL is accessed during the external release stage, and the parameter table carries out full migration every day during gray scale. The flow of the write operation of the parameter table is shown in fig. 5.
And aiming at the read operation of the parameter table, judging whether the user accesses MySQL or DB2 according to the result of the white list user routing strategy. The flow of the read operation of the parameter table is shown in fig. 4.
3. Newly added user access policy
In the public component, the method isInsertDb () of storing and routing the newly added user information, wherein, by configuring the state of the registered user switch in the database system parameter table, it is determined whether the newly added user data is stored in MySQL or DB2, and the flow chart of the access policy of the newly added user is shown in fig. 6.
In a possible implementation manner, the embodiment of the application further relates to a transaction layer control strategy, in which the gray level switching table is used in cooperation with the switch and the routing strategy, so that access control can be performed at the transaction layer, when the data table is switched, the user is prevented from accessing the transaction corresponding to the data table writing operation, the access control can be accurate to control the table dimension and the user dimension, namely, transaction blocking can be performed on switching of the whole table, and transaction blocking can be performed on migration of single user data, so that non-stop distributed database migration is realized.
For a transaction layer control strategy, before data migration, a master control switch is in a rollback state, a change-over switch is not started, and database access is DB2; in the gray level stage, changing the master control switch into gray level; when data is migrated, the change-over switch is in the process of switching, the list switching information list is not ready, the user mark is in the process of switching, the parameter list and the user list (namely white list users) can carry out transaction blocking writing at a transaction layer when carrying out transaction, at the moment, data is not allowed to be written into a database, and only database data migration is carried out; when the database migration is completed, the table switching information is ready, the user identification is switched, the switching switch is switched to be completed, at the moment, the transaction layer does not block writing any more, and the white list user data access route is changed from DB2 to MySQL. In the formal external stage, the master control switch is changed into external release. The whole process can realize the non-stop distributed database migration, only in the data migration process, part of transactions are blocked temporarily, and the data migration is completed to normally provide services for users, so that the influence on the users is greatly reduced, and the customer experience is greatly improved. The transaction layer control flow is shown in fig. 7.
According to the embodiment of the application, through the application layer flow control strategy, the distributed database migration application transformation and the conventional service demand are combined, the rapid iteration, parallel version following and service demand management strategy is adopted, the front-end treatment, application production and database switching are completed through 20 iterations, no influence on the construction of heavy-point projects such as daily business projects is realized, and meanwhile, stable migration of 6000 ten thousand users and 14 hundred million stock data of electronic channels is completed, no perception on clients is realized during migration, and no external service is realized.
The embodiment of the application has the following specific advantages:
1) No stop and no perception to clients
The data migration strategy of gray scale- > small batch- > large batch- > full-scale without stopping machine is adopted, and data table routing strategy control is carried out on an application layer during data migration of a database, so that stable data migration is completed, external service is not stopped during migration, and no perception is realized for a client.
2) No influence on business is caused by demand control
The distributed database migration application transformation is combined with the conventional business demand project in a double-line manner, and the rapid iteration, parallel version following and business demand control strategies are adopted, so that the pretreatment, the application production and the database switching are completed through multiple iterations, and no influence is caused to the construction of heavy-point projects such as business daily projects and the like.
3) Gray release, reduced production event generation
The gray level release mechanism adopts a strategy of fully verifying in advance and then extrapolating the wide range, thereby realizing zero event production.
An example of a practical application of the embodiment of the present application is given below, specifically:
In 2019, the electronic channel is applied to the private business processing platform system by the bank software development center, so as to improve the system performance, reduce the bottleneck of single-point database dependence, plan to migrate the database from DB2 to MySQL, ensure the data reliability and controllable risk for improving the user experience, and apply the application layer flow control strategy to modify the private business processing platform system and apply the application layer flow control strategy for 9 months-12 for multiple times, and the database migration is carried out, so that the production plan is shown in fig. 8.
1. Description of the production phase
(1) Rollback phase: only DB2 was accessed 3 days before 11 months in 2019.
(2) Gray scale phase: 11.3.2019 to 3.3 12 months 1 in 2019. Migration table DB2, mySQL parallel access.
(3) Formally outward stage: the migration table only accesses MySQL after 12 months 1 2019.
Wherein, the two-way support of the DB2 database and MySQL database persists until month 1 in 2020.
2. Data table access specification
(1) User table: only DB2 was accessed before 11 months 3 days 2019. The white list switching and the flow switching are started on the 11 th and 3 rd days of 2019, the white list switching users and the flow switching users access MySQL, and other users access DB2. The new registered user gray period white list user is registered to MySQL, and other users are registered to DB2. The newly registered user starts all switches to MySQL.
(2) Parameter table: only DB2 was accessed before 11 months and 10 days in 2019. Beginning at 11.2019 and 10.11.10, the parameter class parameter table DB2 and MySQL are written in double, mySQL is queried, and the user class parameter table only operates MySQL. The formal external and external parameter tables only operate MySQL.
3. Production effect
The electronic channel 6000 ten thousand users and 14 hundred million stock data are stably migrated, external service is not stopped during migration, no perception is realized to clients, meanwhile, a gray level release mechanism of fully verifying in advance and then extrapolating a broad policy is adopted, so that a formal external production 0 event is realized, an application layer flow control policy for carrying out distributed database migration without stopping is used for enabling the database migration project to be a case that the distributed database is successfully migrated and is not stopped by the first bank in the industry, and the policy has a certain guiding value for the distributed database migration of the same industry system.
Fig. 9 is a schematic structural diagram of a data migration apparatus according to another embodiment of the present application, as shown in fig. 9, the apparatus 900 may include: a determining module 901 and a processing module 902, wherein:
the determining module 901 is configured to determine a user data table to be migrated from the original database, where the user data table to be migrated is all or part of the user data tables in the original database;
The processing module 902 is configured to migrate, based on a preset routing access policy, a gray level switching table and a switching policy, a user data table to be migrated from an original database to a target database, and control data access of the user data table to be migrated in the target database.
In one possible implementation manner, the gray level switching table and the switching strategy comprise a gray level switching table, a user switching table, a table switching information table and a transaction code and table correspondence table; wherein,
The gray level switch table is used for controlling the data access and data migration of the whole application;
the user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database;
the table switching information table stores ready states of the tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database;
The corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process.
In one possible implementation, the gray scale switch table includes a master switch, a change-over switch, and a new user access switch;
the master control switch is used for controlling the data access of the whole application and comprises an initial state, a gray state and an external release state;
the change-over switch is used for controlling a user or a data table to migrate from the original database to the target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over;
The new user access switch is used for controlling the new user to access the original database or the target database.
In one possible implementation, the data in the gray level switch table, the user switch table, the table switch information table and the transaction code and table correspondence table are all set into the remote dictionary service Redis by a timing task, so that the application reads the data from the Redis and writes the updated data into the Redis in real time when the data is updated.
In one possible implementation, when any one of the gray level switch table, the user switching table, the table switching information table and the transaction code and table correspondence table includes key user information, adding an attribute representing the user switching state to any one table for controlling database access of single user data;
wherein the key user information includes at least one of:
User number, mailbox, certificate number, security authentication mobile phone number, login mobile phone number, subscription account number, core client number, machine code, authentication system OpenID;
the user switching state includes non-switching, switching and switched.
In one possible implementation, the method further comprises a classification module; the classification module is used for:
And dividing the user data table to be migrated into a user table and a parameter table according to whether the user data table to be migrated comprises key user information and an application scene of the user data table to be migrated.
In one possible implementation, the routing access policies include a user table access policy and a parameter table access policy;
the user table access strategy is used for controlling a user to access the original database or the target database, wherein if the master control switch is gray and the user identification is switched or the master control switch is issued outwards, the user is controlled to access the target database, otherwise, the user is controlled to access the original database;
The parameter table access strategy is used for controlling a user to access the original database or the target database, wherein if the parameter table state is that the external release or the gray period access mark is the target database, the user is controlled to access the target database, otherwise, the user is controlled to access the original database.
In one possible implementation, the routing access policy further includes an added user access policy;
the newly added user access strategy is used for determining whether the data of the newly added user is stored in the target database or the original database by configuring the state of a switch of the registered user in the database system parameter table.
According to the device provided by the embodiment of the application, through the route access strategy, the gray level switching table and the switching strategy, the data can be stably migrated through the route access strategy control of the data table in the application layer during the data migration, external service is not stopped during the migration, the influence of the data migration in the database on the client in production is reduced, the condition of long-time transaction blocking caused by the shutdown of the application server is avoided, and the data migration which is not perceived by the client is realized; through the gray level release mechanism of the gray level switching table and the switching strategy, the full verification is carried out inside the gray level switching table and the switching strategy, and then the external popularization is carried out, so that the occurrence of production events is greatly reduced.
It should be noted that, this embodiment is an apparatus embodiment corresponding to the above-mentioned method embodiment, and this embodiment may be implemented in cooperation with the above-mentioned method embodiment. The related technical details mentioned in the above method embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the related technical details mentioned in the present embodiment may also be applied in the above-described method item embodiments.
Another embodiment of the present application provides an electronic device, as shown in fig. 10, an electronic device 1000 shown in fig. 10 includes: a processor 1001 and a memory 1003. The processor 1001 is coupled to the memory 1003, such as via a bus 1002. Further, the electronic device 1000 may also include a transceiver 1004. It should be noted that, in practical applications, the transceiver 1004 is not limited to one, and the structure of the electronic device 1000 is not limited to the embodiment of the present application.
The processor 1001 is applied to the embodiment of the present application, and is configured to implement the functions of the determining module and the processing module shown in fig. 9. The transceiver 1004 includes a receiver and a transmitter.
The processor 1001 may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various exemplary logic blocks, modules and circuits described in connection with this disclosure. The processor 1001 may also be a combination that implements computing functionality, such as a combination comprising one or more microprocessors, a combination of a DSP and a microprocessor, or the like.
Bus 1002 may include a path to transfer information between the components. Bus 1002 may be a PCI bus, an EISA bus, or the like. The bus 1002 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 10, but not only one bus or one type of bus.
The memory 1003 may be, but is not limited to, ROM or other type of static storage device, RAM or other type of dynamic storage device, which can store static information and instructions, EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disc, etc.), magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
The memory 1003 is used for storing application code for executing the inventive arrangements and is controlled for execution by the processor 1001. The processor 1001 is configured to execute application code stored in the memory 1003 to implement the actions of the data migration apparatus provided by the embodiment shown in fig. 9.
The electronic device provided by the embodiment of the application comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein when the processor executes the program, the implementation can be realized: determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all or part of the user data tables in the original database; and then, based on a preset route access strategy, a gray level switching table and a switching strategy, migrating the user data table to be migrated from the original database to a target database, and controlling the data access of the user data table to be migrated in the target database.
The embodiment of the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method shown in the above embodiment. The data migration method comprises the steps of performing data migration on a database through a routing access strategy, a gray level switching table and a switching strategy, and performing routing access strategy control on the data table at an application layer during data migration, so that stable data migration can be completed, external service is not stopped during migration, influence of data migration in the database on a client in production is reduced, long-time transaction blocking caused by shutdown of an application server is avoided, and data migration which is not perceived by the client is realized; through the gray level release mechanism of the gray level switching table and the switching strategy, the full verification is carried out inside the gray level switching table and the switching strategy, and then the external popularization is carried out, so that the occurrence of production events is greatly reduced.
The computer readable storage medium provided by the embodiments of the present application is applicable to any one of the embodiments of the above method.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.
The foregoing is only a partial embodiment of the present application, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present application, and such modifications and adaptations are intended to be comprehended within the scope of the present application.
Claims (8)
1. A method of data migration, comprising:
Determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all user data tables or part of user data tables in the original database;
Based on a preset route access strategy, a gray level switching table and a switching strategy, migrating the user data table to be migrated from the original database to a target database, and controlling the data access of the user data table to be migrated in the target database;
the gray level switching table and the switching strategy comprise a gray level switching table, a user switching table, a table switching information table and a transaction code and table corresponding relation table; wherein,
The gray switch table is used for controlling data access and data migration of the whole application;
The user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database;
The table switching information table stores ready states of user tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database;
the corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process;
the gray level switch table comprises a master control switch, a change-over switch and a newly added user access switch;
The master control switch is used for controlling data access of the whole application and comprises an initial state, a gray state and an external release state;
the change-over switch is used for controlling a user or a data table to migrate from an original database to a target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over;
the new user access switch is used for controlling the new user to access the original database or the target database.
2. The method of claim 1, wherein the data in the gray-scale switch table, the user switch table, the table switch information table, and the transaction code to table correspondence table are each clocked into a remote dictionary service, dis, by a timing task such that the application reads data from the dis and writes updated data to the dis in real time as the data is updated.
3. The method of claim 1, wherein when any one of the gray scale switch table, the user switch table, the table switch information table, and the transaction code and table correspondence table includes key user information, adding an attribute characterizing a user switch state to the any one table for controlling database access of a single piece of user data;
wherein the key user information includes at least one of:
User number, mailbox, certificate number, security authentication mobile phone number, login mobile phone number, subscription account number, core client number, machine code, authentication system OpenID;
The user switching state comprises non-switching, switching and switched.
4. A method according to any of claims 1-3, characterized in that before said migrating said user data table to be migrated from said original database to a target database based on a pre-set routing access policy and grey scale switching table and switching policy, further comprising:
and dividing the user data table to be migrated into a user table and a parameter table according to whether the user data table to be migrated comprises key user information and an application scene of the user data table to be migrated.
5. The method of claim 4, wherein the routing access policies include a user table access policy, a parameter table access policy, and an additional user access policy;
The user table access strategy is used for controlling a user to access an original database or a target database, wherein if the master control switch is gray and the user identification is switched or the master control switch is issued outwards, the user is controlled to access the target database, otherwise, the user is controlled to access the original database;
The parameter table access strategy is used for controlling a user to access an original database or a target database, wherein if the parameter table state is that the external release or the gray period access mark is the target database, the user is controlled to access the target database, otherwise, the user is controlled to access the original database;
the newly added user access strategy is used for determining whether the data of the newly added user is stored in the target database or the original database by configuring the state of a switch of the registered user in the database system parameter table.
6. A data migration apparatus, comprising:
the determining module is used for determining a user data table to be migrated from an original database, wherein the user data table to be migrated is all user data tables or part of user data tables in the original database;
the processing module is used for migrating the user data table to be migrated from the original database to a target database based on a preset route access strategy, a preset gray level switching table and a preset switching strategy, and controlling the data access of the user data table to be migrated in the target database;
the gray level switching table and the switching strategy comprise a gray level switching table, a user switching table, a table switching information table and a transaction code and table corresponding relation table; wherein,
The gray switch table is used for controlling data access and data migration of the whole application;
The user switching table stores the identifications of the original database and the target database to be accessed by the user to be switched, and is used for controlling the access of the user-level database;
The table switching information table stores ready states of user tables to be switched, table types and table access routes, and is used for controlling database access of data table levels, wherein the ready states comprise non-ready states and ready states, the table types comprise user tables and parameter tables, and the table access routes comprise an original database and a target database;
the corresponding relation table of the transaction code and the table is used for storing the table switching or writing the transaction list in the user switching process;
the gray level switch table comprises a master control switch, a change-over switch and a newly added user access switch;
The master control switch is used for controlling data access of the whole application and comprises an initial state, a gray state and an external release state;
the change-over switch is used for controlling a user or a data table to migrate from an original database to a target database, and comprises a state of not starting to be changed over, a state in the process of being changed over and a state of completing the change-over;
the new user access switch is used for controlling the new user to access the original database or the target database.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1-5 when executing the program.
8. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110333945.5A CN112860670B (en) | 2021-03-29 | 2021-03-29 | Data migration method, data migration device, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110333945.5A CN112860670B (en) | 2021-03-29 | 2021-03-29 | Data migration method, data migration device, electronic equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112860670A CN112860670A (en) | 2021-05-28 |
| CN112860670B true CN112860670B (en) | 2024-10-08 |
Family
ID=75993187
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110333945.5A Active CN112860670B (en) | 2021-03-29 | 2021-03-29 | Data migration method, data migration device, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860670B (en) |
Families Citing this family (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113468150A (en) * | 2021-09-03 | 2021-10-01 | 四川新网银行股份有限公司 | Horizontal segmentation capacity expansion and migration method for payment subscription data |
| CN113722297A (en) * | 2021-09-14 | 2021-11-30 | 首约科技(北京)有限公司 | Order system reconstruction smooth migration method |
| CN114036165A (en) * | 2021-11-10 | 2022-02-11 | 中国建设银行股份有限公司 | Batch switching method and system based on white list |
| CN114020724B (en) * | 2021-11-10 | 2025-07-29 | 中国工商银行股份有限公司 | Instruction library migration method, apparatus, computer device and storage medium |
| CN114218187B (en) * | 2021-11-16 | 2025-03-21 | 中国建设银行股份有限公司 | Data online migration method, device, electronic device and storage medium |
| CN114297181B (en) * | 2021-12-28 | 2025-09-23 | 中国邮政储蓄银行股份有限公司 | Data migration processing method and device |
| CN114416695B (en) * | 2022-01-19 | 2025-05-27 | 中国平安人寿保险股份有限公司 | Data splicing function migration method, device, computer equipment and storage medium |
| CN114780512B (en) * | 2022-03-22 | 2023-05-12 | 荣耀终端有限公司 | Gray release method, system and server |
| CN114936935A (en) * | 2022-05-30 | 2022-08-23 | 中国银行股份有限公司 | Financial transaction processing method and device |
| CN115858489B (en) * | 2022-11-23 | 2024-08-06 | 中电金信软件有限公司 | Transaction processing method, device, computer equipment and medium based on data migration |
| CN116595075A (en) * | 2023-04-18 | 2023-08-15 | 上海中通吉网络技术有限公司 | Method, system and electronic equipment for switching database gray levels |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019133A (en) * | 2017-12-21 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Online data moving method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109359159A (en) * | 2018-09-30 | 2019-02-19 | 深圳前海微众银行股份有限公司 | Distributed storage method, system and device |
-
2021
- 2021-03-29 CN CN202110333945.5A patent/CN112860670B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019133A (en) * | 2017-12-21 | 2019-07-16 | 北京京东尚科信息技术有限公司 | Online data moving method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112860670A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112860670B (en) | Data migration method, data migration device, electronic equipment and computer storage medium | |
| US20220067025A1 (en) | Ordering transaction requests in a distributed database according to an independently assigned sequence | |
| CN111143389B (en) | Transaction execution method and device, computer equipment and storage medium | |
| CN111338766B (en) | Transaction processing method, apparatus, computer equipment and storage medium | |
| US20240211461A1 (en) | Customer-requested partitioning of journal-based storage systems | |
| CN108804112B (en) | Block chain settlement processing method and system | |
| US8935205B2 (en) | System and method of performing snapshot isolation in distributed databases | |
| CN111367989B (en) | Real-time data index calculation system and method | |
| JP2023546249A (en) | Transaction processing methods, devices, computer equipment and computer programs | |
| CN101964820B (en) | Method and system for keeping data consistency | |
| US9990225B2 (en) | Relaxing transaction serializability with statement-based data replication | |
| US20140149527A1 (en) | Slave Side Transaction ID Buffering for Efficient Distributed Transaction Management | |
| WO2010104902A2 (en) | Composite hash and list partitioning of database tables | |
| US9971822B1 (en) | Replicated state management using journal-based registers | |
| US11797523B2 (en) | Schema and data modification concurrency in query processing pushdown | |
| US10489356B1 (en) | Truncate and append database operation | |
| US20250068621A1 (en) | Systems and methods for application data servers | |
| US20230376479A1 (en) | Schema and data modification concurrency in query processing pushdown | |
| US11714745B1 (en) | Software testing in parallel with different database instances | |
| US12242453B2 (en) | Data processing method and apparatus, electronic device, storage medium, and program product | |
| US11537508B1 (en) | Software testing in parallel threads with a record-locking database | |
| US8639657B2 (en) | Reorganizing table-based data objects | |
| US20110320474A1 (en) | Availability of Database Objects During Database Operations | |
| US10664361B1 (en) | Transactionally consistent backup of partitioned storage | |
| WO2017087536A1 (en) | Multi-version removal manager |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |