CN110858194A - Method and device for expanding database - Google Patents

Abstract

The invention discloses a database expansion method and device, and relates to the technical field of computers. A specific implementation of the method includes: acquiring database monitoring information, and determining whether the database needs to be expanded according to preconfigured expansion rules and the database monitoring information; if the database needs to be expanded, according to the current database classification The slice rule establishes at least one new library table; according to the preset routing read and write rules, the original data and newly added data of the database are respectively written into the corresponding library table; wherein, the library table includes the new library surface. After determining that the database needs to be expanded, the method adds a new database table according to the current database sharding rules, and writes data into the corresponding database table based on the routing read and write rules, and can realize automatic database sharding and data migration without downtime for deployment .

Description

Method and device for database expansion

technical field

The invention relates to the field of computers, and in particular, to a method and device for database expansion.

Background technique

In the process of developing enterprise-level applications, the retrieval performance of relational databases will drop sharply when the amount of data is larger than a certain amount. When you need to operate on massive data, if you save all the data in one data table, it will often exceed the threshold that the data table can store. In addition, when too many concurrent requests access the same database, the response of the database will also change. slow. In the prior art, concurrent requests are distributed to different databases and data tables through the strategy of sub-database and sub-table, thereby solving the problems of poor database retrieval performance and slow database response caused by large data volume and high concurrency.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art:

(1) In the process of implementing sub-database and sub-table, each time adding a database and adding a table needs to modify the database file, and it is necessary to stop the operation of the server and redeploy the application on the server, which increases the risk of the online business system;

(2) After the database is added and the table is added, the data volume of the original database table is large, and the data volume of the new database table is small, resulting in unbalanced data storage;

(3) In the process of implementing sub-database and sub-table, manual data migration is required, which consumes a lot of labor costs, and also affects the correctness of data during the data migration process.

SUMMARY OF THE INVENTION

In view of this, an embodiment of the present invention provides a method and device for database expansion. After determining that the database needs to be expanded, a new database table is added according to the current database fragmentation rules, and data is written into the corresponding database table based on routing read-write rules. , automatic database sharding and data migration can be realized without downtime for deployment.

To achieve the above object, according to an aspect of the embodiments of the present invention, a method for expanding the capacity of a database is provided.

A method for database expansion according to an embodiment of the present invention includes: acquiring database monitoring information, and determining whether to expand the database according to preconfigured expansion rules and the database monitoring information; if the database needs to be expanded, Then at least one new database table is established according to the current database fragmentation rules; according to the preset routing read and write rules, the original data and newly added data of the database are respectively written into the corresponding library table; wherein, the library table Include the new library table.

Optionally, the database monitoring information includes at least one of the following: data storage capacity, connection utilization, and data volume of a single table; the capacity expansion rule includes at least one of the following: when the data storage capacity is greater than a first threshold, The database needs to be expanded; when the connection utilization rate is greater than the second threshold, the database needs to be expanded; and when the data volume of the single table is greater than the third threshold, the database needs to be expanded.

Optionally, the database fragmentation rule is provided with the number of expanded databases and data tables; the establishing at least one new database table according to the current database fragmentation rule includes: according to the number of the databases, the data in the database The number of tables, and the number of expanded databases and data tables set in the current database sharding rules, create a corresponding number of databases and data tables.

Optionally, according to the preset routing reading and writing rules, the original data and the newly added data of the database are respectively written into the corresponding library tables, including: according to the preset routing reading and writing rules, determining the The library table to which each piece of data in the database belongs; filter the original data in the original library table, write the data obtained by filtering into the new library table of the attribution, and record the current filtering start time; determine whether the original data is The filtering is completed. If the filtering is not completed, the next filtering start time is determined according to the current filtering start time and the filtering interval, so as to continue the next filtering until all the original data are filtered; the newly added data is written into into the attribution of the library table.

Optionally, the routing read-write rule is obtained based on any one of the following algorithms: a ring hash algorithm, a hash algorithm, and a modulo algorithm.

Optionally, the method further includes: judging whether the capacity expansion is successful, and if the capacity expansion is successful, updating the capacity expansion rule.

To achieve the above object, according to another aspect of the embodiments of the present invention, an apparatus for expanding the capacity of a database is provided.

A database capacity expansion device according to an embodiment of the present invention includes: a determination module for acquiring database monitoring information, and determining whether the database needs to be expanded according to a preconfigured capacity expansion rule and the database monitoring information; a building module for using If the database needs to be expanded, then at least one new database table is established according to the current database fragmentation rules; the writing module is used to convert the original data and newly added data of the database according to the preset routing read and write rules. are respectively written into the corresponding library table; wherein, the library table includes the new library table.

Optionally, the database monitoring information includes at least one of the following: data storage capacity, connection utilization, and data volume of a single table; the capacity expansion rule includes at least one of the following: when the data storage capacity is greater than a first threshold, The database needs to be expanded; when the connection utilization rate is greater than the second threshold, the database needs to be expanded; and when the data volume of the single table is greater than the third threshold, the database needs to be expanded.

Optionally, the database fragmentation rule is provided with the number of expanded databases and data tables; the establishment module is also used for: according to the number of the databases, the number of data tables in the database, and the current database partition. According to the number of expanded databases and data tables set in the slice rule, create a corresponding number of databases and data tables.

Optionally, the writing module is further configured to: determine the library table to which each piece of data in the database belongs according to a preset routing read-write rule; filter the original data in the original library table, and filter the The obtained data is written in the described new library table of attribution, and the current filtering start time is recorded; it is judged whether the filtering of the original data is completed, if not, the next filtering start time is determined according to the current filtering start time and the filtering interval. time, so as to continue the next filtering until all the filtering of the original data is completed; and write the newly added data into the attributable library table.

Optionally, the routing read-write rule is obtained based on any one of the following algorithms: a ring hash algorithm, a hash algorithm, and a modulo algorithm.

Optionally, the apparatus further includes: an update module configured to determine whether the capacity expansion is successful, and if the capacity expansion is successful, update the capacity expansion rule.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, an electronic device is provided.

An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, The one or more processors are caused to implement a method for expanding the capacity of a database according to an embodiment of the present invention.

To achieve the above object, according to yet another aspect of the embodiments of the present invention, a computer-readable medium is provided.

A computer-readable medium according to an embodiment of the present invention stores a computer program thereon, and when the program is executed by a processor, a method for expanding a database according to an embodiment of the present invention is implemented.

One embodiment of the above invention has the following advantages or beneficial effects: after it is determined that the database needs to be expanded, a new database table is added according to the current database fragmentation rules, and data is written into the corresponding database table based on routing read-write rules, without downtime Deployment can realize automatic sub-database sub-table and data migration; pre-configure the content contained in the expansion rules according to business requirements, and trigger the expansion process after the database monitoring information satisfies the expansion rules to realize automatic expansion; according to the current database fragmentation The number of databases and data tables set in the rules, as well as the number of databases and data tables corresponding to the current business system, create new databases and data tables, and realize automatic database and table addition; based on routing read and write rules, determine each entry in the database The database table to which the data belongs is filtered and copied according to the determined database table to realize automatic data migration; the routing read and write rules are implemented through the ring hash algorithm, which can ensure the balanced distribution of data to each database table; Correctly and conveniently determine whether the expanded database needs to be expanded again.

Further effects of the above non-conventional alternatives will be described below in conjunction with specific embodiments.

Description of drawings

The accompanying drawings are used for better understanding of the present invention and do not constitute an improper limitation of the present invention. in:

1 is a schematic diagram of main steps of a method for database expansion according to an embodiment of the present invention;

FIG. 2 is a schematic flowchart of a main flow of a method for database expansion according to an embodiment of the present invention;

3 is a schematic diagram of the results before fission from four libraries and four tables to four libraries and eight tables according to an embodiment of the present invention;

4 is a schematic diagram of the results after fission from four libraries and four tables to four libraries and eight tables according to an embodiment of the present invention;

5 is a schematic diagram of the results after fission from four pools and four tables to eight pools and eight tables according to an embodiment of the present invention;

6 is a schematic diagram of the main steps of writing original data and newly added data of a database into a corresponding library table according to an embodiment of the present invention;

7 is a schematic diagram of an implementation principle of writing data from a data table into a new library table according to an embodiment of the present invention;

8 is a schematic diagram of main modules of an apparatus for database expansion according to an embodiment of the present invention;

9 is a schematic diagram of main modules of a database capacity expansion system according to an embodiment of the present invention;

FIG. 10 is an exemplary system architecture diagram to which an embodiment of the present invention can be applied;

FIG. 11 is a schematic structural diagram of a computer device suitable for implementing the electronic device according to the embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

FIG. 1 is a schematic diagram of main steps of a method for database expansion according to an embodiment of the present invention. As shown in FIG. 1 , the method for database expansion according to the embodiment of the present invention mainly includes the following steps:

Step S101: Obtain database monitoring information, and determine whether the database needs to be expanded according to a preconfigured expansion rule and the database monitoring information. Wherein, the database monitoring information includes at least one of the following: data storage capacity, connection utilization, and data volume of a single table; the capacity expansion rule includes at least one of the following: when the data storage capacity is greater than a first threshold, it is necessary to The database is expanded; when the connection utilization rate is greater than the second threshold, the database needs to be expanded; and when the data volume of the single table is greater than the third threshold, the database needs to be expanded. When the database monitoring information satisfies any of the above-mentioned expansion rules, the expansion process of step S102 is triggered.

Step S102: If the database needs to be expanded, at least one new database table is established according to the current database fragmentation rules. The current database fragmentation rule is set with the number of expanded databases and the number of data tables. According to the number of databases, the number of data tables in the database, and the number of expanded databases and the number of data tables set in the current database sharding rule, a corresponding number of databases and data tables are newly created. For example, assuming the original two databases and two tables, the current database sharding rules need to expand to two databases and four tables, then create two new data tables; assuming the original two databases and two tables, the current database sharding rules need to be expanded to four databases Four tables, then create two new databases and two data tables.

Step S103: Write the original data and the newly added data of the database into the corresponding library table respectively according to the preset routing reading and writing rules; wherein, the library table includes the new library table. The routing read-write rule is obtained based on any one of the following algorithms: a ring hash algorithm, a hash algorithm and a modulo algorithm. The specific writing process is as follows: according to the preset routing read and write rules, determine the library table to which each piece of data in the database belongs; filter the original data in the original library table, and write the filtered data into the home table. In the new library table, record the current filtering start time; determine whether the original data is filtered and complete, if not, then determine the next filtering start time according to the current filtering start time and the filtering interval, to continue the next filtering , until all the original data is filtered; write the newly added data into the attributable library table. Recording the current filtering start time can ensure the correctness of the data migration process.

FIG. 2 is a schematic flowchart of a main flow of a method for database expansion according to an embodiment of the present invention. As shown in FIG. 2, the method for database expansion according to the embodiment of the present invention mainly includes the following steps:

Step S201: Store database information and business system information in a management and control platform, and configure expansion rules and database fragmentation rules on the management and control platform. The database information includes domain name, port number, user name and password. The business system information includes business system basic information and configuration information, and the business system basic information includes business system identifiers (such as globally unique English names), Chinese names, affiliation departments, R&D leaders, contact numbers, email addresses, and project teams. Member list; the configuration information is used to configure databases and data tables, including business system identifiers (such as globally unique English names), domain names, user names and passwords. The number of expanded databases and the number of data tables are set in the database fragmentation rule.

The capacity expansion rule includes at least one of the following: when the data storage capacity is greater than a first threshold, the database needs to be expanded; when the connection utilization rate is greater than a second threshold, the database needs to be expanded; And when the amount of data in the single table is greater than the third threshold, the database needs to be expanded. The first threshold may be the maximum data storage capacity multiplied by a preset first ratio. For example, if the maximum data storage capacity is 10G and the first ratio is 80%, the first threshold may be 8G; the second threshold may be The maximum number of connections or the current number of connections is multiplied by a preset second ratio. For example, if the maximum number of connections is 100 and the second ratio is 80%, the second threshold is 80; the third threshold may be the maximum data volume of a single table multiplied by The third ratio is preset, for example, the maximum data volume of a single table is 500W pieces of data, and the third ratio is 80%, then the third threshold is 400W pieces of data. The first ratio, the second ratio and the third ratio are determined according to actual needs, and the three may be the same or different.

Step S202: The management control platform synchronizes the database information and the database fragmentation rules to a data registration center, and synchronizes the database information and the expansion rules to a data monitoring center. In an embodiment, the service system acquires configuration information from the configuration node, so as to store the configuration information in the data registration center. The business system monitors the configuration information, and if the configuration information is modified, stores the modified configuration information in the data registration center. The configuration node can be implemented using Zookeeper (a distributed, open source distributed application coordination service), Etcd (a distributed key-value storage system), etc., and the configuration information includes database information and database sharding rules.

After the configuration information is modified, the configuration node will synchronize the latest configuration information to the configuration center. Every time the application of the business system will listen to the notification of the configuration center, when the monitoring changes, the modified configuration information will be synchronized to the local cache, so that the configuration information can be refreshed without restarting, and then the database can be dynamically increased or decreased and modified Database sharding rules.

Step S203: Start the application of the business system to obtain the database information and database fragmentation rules corresponding to the business system from the data registration center. After the application starts, it is necessary to judge whether the obtained database information and database sharding rules are correct, that is, whether the application starts abnormally and whether it can access the database normally. If the application starts normally and can access the database normally, it means that the obtained data is correct. In a preferred embodiment, when the application is just deployed to the business system, the business system needs to pull the configuration information from the data registration center.

Step S204: the management control platform monitors the database through the data monitoring center. According to the first to third thresholds set in the management control platform, the data monitoring center monitors the data storage capacity, connection utilization rate and single-table data volume of the database respectively, and feeds back the database monitoring information obtained by monitoring to the management control platform. The content monitored by the data monitoring center is not limited to the above three, and the specific monitoring content is determined by the user, such as database performance, server performance, etc. can also be monitored.

Step S205: The management control platform determines whether the database needs to be expanded according to the database monitoring information obtained by monitoring and the capacity expansion rule, and if not, executes step S204; if necessary, executes step S206. When the data storage capacity of the database is greater than the first threshold, or the connection utilization rate is greater than the second threshold, or the data volume of a single table is greater than the third threshold, it indicates that the database needs to be expanded.

Step S206: The management control platform suspends monitoring of the database, and establishes at least one new database table according to the new database fragmentation rule. Wherein, the new database fragmentation rule, that is, the current database fragmentation rule, needs to be obtained from the configuration information. The establishment of at least one new database table according to the current database fragmentation rule includes: according to the number of the database, the number of data tables in the database, and the number and data of the expanded database set in the current database fragmentation rule The number of tables, create a corresponding number of databases and data tables. For example, if the original two databases and two tables are expanded into two databases and four tables, then two new data tables need to be created; if the original two databases and two tables are expanded into four databases and four tables, then two new databases and two new data tables need to be created. data table.

The new database fragmentation rules involve three fission modes: fission of data table, fission of database library, and fission of library table. Among them, fission refers to the shunting of data from one source to multiple sources. The fission of the data table is suitable for the situation that the amount of data is large, and the data concurrency can meet the current business scenario; the fission of the database database is suitable for the situation that the data concurrency cannot meet the current business scenario; the database table all occurs Fission is suitable for situations where the amount of data is large and the amount of data concurrency cannot meet the current business scenario. The specific method for fission should be determined according to the amount of data and the amount of data concurrency. The number of databases and data tables generated by fission can be defined according to business needs.

The large amount of data means that the current amount of data is greater than the preset third threshold. For example, the third threshold is 500W pieces of data. If the current amount of data is greater than 500W pieces of data, fission of the data table is performed. FIG. 3 and FIG. 4 are respectively the results before and after the fission from four libraries and four tables to four libraries and eight tables according to an embodiment of the present invention. As shown in Figure 3, the four databases before fission are DB1-DB4, and each database has four data tables, Tb0-Tb3. After the fission of the data table, as shown in Figure 4, the database remains unchanged, and each database has eight data tables respectively Tb0-Tb7.

The data concurrency amount that can satisfy the current business scenario means that the current data concurrency amount (corresponding to the connection utilization rate) is greater than the second threshold, for example, the second threshold is 500. If the current data concurrency amount is greater than 500, the database is fissioned. FIG. 5 is the result of fission from four pools and four tables to eight pools and eight tables according to an embodiment of the present invention. The four databases and four tables before fission are still as shown in Figure 3, including four databases DB1-DB4, and four data tables Tb0-Tb3 in each database. After the fission of the database, as shown in Figure 5, four databases are added, that is, a total of eight databases are DB1-DB8, and each database has a total of four data tables Tb0-Tb3.

Step S207: The management and control platform respectively writes the original data and the newly added data of the database into the corresponding library table according to the preset routing read and write rules. Wherein, the library table includes the original library table and the new library table. The following is an example of how the routing read and write rules are obtained based on the ring hash algorithm. The realization principle of the ring hash algorithm is: calculate the key value (Key) of the original data, the new data and the database through the hash function, and map the Key of the original data and the new data and the Key of the database to a 2 respectively. On the ring of ^32; then search clockwise from the position of the key of the original data and the new data on the ring, and store the key in the first database. The above algorithm evenly distributes the databases on the ring in a clockwise direction, and the keys of the original data and the new data will belong to a database on the ring. The core code of the algorithm is as follows:

Step S208: The management control platform determines whether the expansion is successful, and if not, executes step S209; if yes, executes step S211. The management control platform will record the rollback time point during the expansion process. If the expansion is unsuccessful, the original data of the database can be restored to the rollback time point through the rollback mechanism to re-expand the capacity.

Step S209: The management control platform restores the original data of the database to a rollbackable time point through a rollback mechanism, and executes step S210.

Step S210: The management control platform rewrites the original data corresponding to the rollback time point into the corresponding new library table, and executes step S208.

Step S211: The management control platform updates the expansion rules to obtain new expansion rules, synchronizes the new database fragmentation rules to the data registration center, and synchronizes the new expansion rules to the data monitoring center. After the expansion process, the data storage capacity and/or the maximum connection value of the database will increase, so the first threshold and/or the second threshold in the expansion rule need to be updated.

Step S212: The business system pulls the new data fragmentation rule from the data registration center to initiate access to the expanded database according to the new data fragmentation rule. Through the above processing, automatic data monitoring, automatic expansion, automatic data migration, and no need to redeploy applications are completed, maintenance costs are reduced, and R&D efficiency and business processing capabilities are improved.

FIG. 6 is a schematic diagram of main steps for writing original data and newly added data of a database into a corresponding library table according to an embodiment of the present invention. As shown in Figure 6, step S207 includes:

Step S601: Determine the library table to which each piece of data in the database belongs according to a preset routing read-write rule. Taking a MySQL (which is an open-source relational database management system) database with one master data table and two slave data tables as an example, one of the slave data tables of the master data table is offline processed, and the slave data table of the master data table is processed offline. The slave data table after the line determines the library table to which each data in the slave data table belongs according to the routing read and write rules.

Step S602: Filter the original data in the original library table, write the filtered data into the new library table to which it belongs, and record the current filtering start time. According to the determined library table, filter the corresponding data from the data table and copy it to the new library table, and delete the original data from the data table to maintain the consistency of the data.

FIG. 7 is a schematic diagram of an implementation principle of writing data from a data table into a new library table according to an embodiment of the present invention. As shown in Figure 7, suppose there are two databases DB0 and DB1, DB0 has master data tables Tb00 and Tb01, master data table Tb00 has two slave data tables Tb0, and master data table Tb01 has two slave data tables Tb1; DB1 There are master data tables Tb00 and Tb01, the master data table Tb00 has two slave data tables Tb0, and the master data table Tb01 has two slave data tables Tb1. According to business requirements, the two databases and two tables need to be expanded into two databases and four tables, that is, DB0 is expanded into four data tables Tb0, Tb1, Tb2, Tb3; DB1 is expanded into four data tables Tb0, Tb1, Tb2, Tb3. After one of the slave data tables of each master data table is offline processed, the data of the slave data table is filtered and copied to the data tables of DB0 and DB1.

Step S603: Determine whether all the filtering of the original data is completed, and if so, end the process; if not, determine the next filtering start time according to the current filtering start time and filtering interval, so as to continue the next filtering until All the raw data are filtered. When the filtering is completed, the data in the library table is consistent with the original data and the newly added data. In the embodiment, after the filtering is completed, the offline slave database table is brought back online, and data is synchronized from the corresponding master data table.

Step S604: Write the newly added data into the attributable library table. In this embodiment, step S604 is used to write the newly added data into the corresponding library table, and steps S602 to S603 are used to migrate the original data to the corresponding new library table. The embodiment does not limit the execution order of step S604 , which can be executed between step S601 and step S602.

It is assumed that the application of the business system initializes the order data into two tables and two databases according to the order ID (orderId). Due to the needs of business development, the database tables need to be expanded into two databases and four tables. Taking the ring hash algorithm as an example, the principle of how to write the original data and newly added data of the database into the corresponding database table in the above example will be described.

Suppose there are 16 pieces of order data, and the last two-digit numbers of each order data are 00, 01, . As shown in Table 1, the distribution results obtained by writing the order data into the two databases and four tables are shown in Table 2 (each order data in the following distribution results is evenly distributed, just for the convenience of description, the actual distribution results are can be obtained by the hash algorithm).

Table 1 shows the order data allocation results of two databases and two tables

library name Table Name orderId DB0 Table_00 00, 04, 08, 12 DB0 Table_01 01, 05, 09, 13 DB1 Table_00 02, 06, 10, 14 DB1 Table_01 03, 07, 11, 15

Table 2 shows the order data allocation results of two databases and four tables

library name Table Name orderId DB0 Table_00 00, 08 DB0 Table_01 01, 09 DB0 Table_02 02, 10 DB0 Table_03 03, 11 DB1 Table_00 04, 12 DB1 Table_01 05, 13 DB1 Table_02 06, 14 DB1 Table_03 07, 15

However, before all the original data of the original database table is migrated, new data will be inserted, and the new data inserted at this time will be written into the database table according to the new database sharding rules. Since the original data is not completely migrated to the new database and new table, it will be read from the original database table and the new database table when reading. For example, the order data with the order ID of 15, according to the processing logic of two databases and two tables, the order data will be written to the 01 table (Table_01) of database 1 (DB1), but it will be written to the database after the database is divided into the table. 1 (DB1) in the 03 table (Table_03). During the data migration process, the order data allocation results for inserting new data are shown in Table 3.

Table 3 is the result of order data assignment for inserting new data

library name Table Name orderId DB0 Table_00 00, 04, 08, 12 DB0 Table_01 01, 05, 09, 13 DB0 Table_02 DB0 Table_03 DB1 Table_00 02, 06, 10, 14 DB1 Table_01 03, 07, 11, 15 DB1 Table_02 DB1 Table_03 15

During the data migration process (that is, when the data migration has not been completed), if data needs to be queried in the database, the generated database table corresponding to the original database sharding rule and the new database sharding rule is queried. Still taking the order data whose order ID is 15 as an example, during the data migration process, it is necessary to query Table_01 of DB1 and Table_03 of DB1.

After the data migration is completed, if you need to query the data in the database, you can query the generated database table corresponding to the new database sharding rule. Still taking the order data whose order ID is 15 as an example, after the data migration is completed, only Table_03 of DB1 needs to be queried.

It can be seen from the database expansion method of the embodiment of the present invention that, after determining that the database needs to be expanded, a new database table is added according to the current database fragmentation rules, and data is written into the corresponding database table based on the routing read and write rules, without downtime Deployment can realize automatic sub-database sub-table and data migration; pre-configure the content contained in the expansion rules according to business requirements, and trigger the expansion process after the database monitoring information satisfies the expansion rules to realize automatic expansion; according to the current database fragmentation The number of databases and data tables set in the rules, as well as the number of databases and data tables corresponding to the current business system, create new databases and data tables, and realize automatic database and table addition; based on routing read and write rules, determine each entry in the database The database table to which the data belongs is filtered and copied according to the determined database table to realize automatic data migration; the routing read and write rules are implemented through the ring hash algorithm, which can ensure the balanced distribution of data to each database table; Correctly and conveniently determine whether the expanded database needs to be expanded again.

FIG. 8 is a schematic diagram of main modules of an apparatus for database expansion according to an embodiment of the present invention. As shown in FIG. 8 , an apparatus 800 for database expansion according to an embodiment of the present invention mainly includes:

The determining module 801 is configured to obtain database monitoring information, and determine whether the database needs to be expanded according to a preconfigured expansion rule and the database monitoring information. Wherein, the database monitoring information includes at least one of the following: data storage capacity, connection utilization, and data volume of a single table; the capacity expansion rule includes at least one of the following: when the data storage capacity is greater than a first threshold, it is necessary to The database is expanded; when the connection utilization rate is greater than the second threshold, the database needs to be expanded; and when the data volume of the single table is greater than the third threshold, the database needs to be expanded. When the database monitoring information satisfies any of the above-mentioned expansion rules, the expansion processing process of the establishment module 802 is triggered.

The establishment module 802 is configured to establish at least one new database table according to the current database fragmentation rules if the database needs to be expanded. The current database fragmentation rule is set with the current number of expanded databases and the number of data tables. According to the number of databases, the number of data tables in the database, and the number of expanded databases and the number of data tables set in the current database sharding rule, a corresponding number of databases and data tables are newly created. For example, assuming the original two databases and two tables, the current database sharding rules need to expand to two databases and four tables, then create two new data tables; assuming the original two databases and two tables, the current database sharding rules need to be expanded to four databases Four tables, then create two new databases and two data tables.

The writing module 803 is configured to write the original data and newly added data of the database into the corresponding library table respectively according to the preset routing reading and writing rules; wherein, the library table includes the new library table. The routing read-write rule is obtained based on any one of the following algorithms: a ring hash algorithm, a hash algorithm and a modulo algorithm. The specific writing process is as follows: according to the preset routing read and write rules, determine the library table to which each piece of data in the database belongs; filter the original data in the original library table, and write the filtered data into the home table. In the new library table, record the current filtering start time; determine whether the original data is filtered and complete, if not, then determine the next filtering start time according to the current filtering start time and the filtering interval, to continue the next filtering , until all the original data is filtered; write the newly added data into the attributable library table. Recording the current filtering start time can ensure the correctness of the data migration process.

In addition, the apparatus 800 for database expansion according to the embodiment of the present invention may further include: an update module (not shown in FIG. 8 ), the module is used to determine whether the expansion is successful, and if the expansion is successful, update the expansion rule.

FIG. 9 is a schematic diagram of main modules of a database capacity expansion system according to an embodiment of the present invention. As shown in FIG. 9 , the database capacity expansion system 900 according to the embodiment of the present invention mainly includes a business system cluster 901, a database cluster 902, a data registration center 903, and a management control platform 904 (ie, a database capacity expansion device 800. The specific implementation functions are as described above. ) and data monitoring center 905.

The business system cluster 901 includes at least one business system, each business system includes at least one business instance, and each business instance receives configuration information of the configuration node, routing read and write rules, and the business instance performs internal SQL (Structured Query) through the Cartesian product algorithm. Language, Structured Query Language) flows and connects with the database cluster through the data engine. The business system obtains configuration information from the configuration node, and stores the configuration information in the data registration center. The Cartesian product algorithm is a mathematical model used to join any two unrelated data tables. Assuming set A={a,b}, set B={0,1,2}, then the Cartesian product of the two sets is {(a,0),(a,1),(a,2),( b,0),(b,1),(b,2)}.

The database cluster 902 includes at least one database, and each database includes at least one data table. Data interaction can be implemented between the business system cluster 901 and the database cluster 902 through DDL (Data Definition Language, data definition language) and DML (Data Manipulation Language, data manipulation language).

The data registration center 903 is used for receiving database information and database sharding rules from the management control platform 904 , and for storing the configuration information from the business system cluster 901 . The database information includes the service system identifier. Data interaction is performed between the data registration center 903 and the business system cluster 901 by means of subscription (Subscribe) and notification (Notify).

The data monitoring center 905 is used for monitoring the database cluster, and feeding back the acquired monitoring database information to the management control platform 904 . The content to be monitored can be the data storage capacity of the database, connection utilization, data volume of a single table, database performance, server performance, etc. Data interaction is performed between the data monitoring center 905 and the database cluster 902 by means of monitoring (Monitor) and notification (Notify).

As can be seen from the above description, after it is determined that the database needs to be expanded, a new database table is added according to the current database fragmentation rules, and data is written into the corresponding database table based on the routing read and write rules, and automatic database partitioning can be realized without downtime for deployment Table sharding and data migration; pre-configure the content contained in the expansion rules according to business requirements, and trigger the expansion process after the database monitoring information satisfies the expansion rules to realize automatic expansion; according to the database and data set in the current database sharding rules The number of tables, as well as the number of databases and data tables corresponding to the current business system, create new databases and data tables, and realize automatic database and table addition; Automatic data migration is realized by filtering and duplicating the determined database table; the routing read and write rules are implemented through the ring hash algorithm, which can ensure the balanced distribution of data to each database table; by updating the expansion rules, the expansion can be determined correctly and conveniently. Whether the subsequent database needs to be expanded again.

FIG. 10 shows an exemplary system architecture 1000 of a method for database expansion or an apparatus for database expansion according to an embodiment of the present invention.

As shown in FIG. 10 , the system architecture 1000 may include terminal devices 1001 , 1002 , and 1003 , a network 1004 and a server 1005 . The network 1004 is a medium used to provide a communication link between the terminal devices 1001 , 1002 , 1003 and the server 1005 . The network 1004 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user can use the terminal devices 1001, 1002, 1003 to interact with the server 1005 through the network 1004 to receive or send messages and the like. Various communication client applications can be installed on the terminal devices 1001 , 1002 and 1003 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, and the like.

The terminal devices 1001, 1002, 1003 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

The server 1005 may be a server that provides various services, for example, a background management server that provides support for shopping websites browsed by the terminal devices 1001 , 1002 and 1003 . The background management server can analyze and process the received product information query request and other data, and feed back the processing results (eg, target push information, product information) to the terminal device.

It should be noted that the method for expanding the database capacity provided by the embodiments of the present application is generally performed by the server 1005 , and accordingly, the apparatus for expanding the database capacity is generally set in the server 1005 .

It should be understood that the numbers of terminal devices, networks and servers in FIG. 10 are only illustrative. There can be any number of terminal devices, networks and servers according to implementation needs.

According to an embodiment of the present invention, the present invention further provides an electronic device and a computer-readable medium.

The electronic device of the present invention comprises: one or more processors; storage means for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more programs Multiple processors implement a method for expanding the capacity of a database according to an embodiment of the present invention.

The computer-readable medium of the present invention stores a computer program thereon, and when the program is executed by the processor, a method for expanding the capacity of a database according to an embodiment of the present invention is implemented.

Referring next to FIG. 11 , it shows a schematic structural diagram of a computer system 1100 suitable for implementing an electronic device according to an embodiment of the present invention. The electronic device shown in FIG. 11 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present invention.

As shown in FIG. 11, a computer system 1100 includes a central processing unit (CPU) 1101, which can be loaded into a random access memory (RAM) 1103 according to a program stored in a read only memory (ROM) 1102 or a program from a storage section 1108 Instead, various appropriate actions and processes are performed. In the RAM 1103, various programs and data necessary for the operation of the computer system 1100 are also stored. The CPU 1101 , the ROM 1102 , and the RAM 1103 are connected to each other through a bus 1104 . An input/output (I/O) interface 1105 is also connected to the bus 1104 .

The following components are connected to the I/O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc. ; and a communication section 1109 including a network interface card such as a LAN card, a modem, and the like. The communication section 1109 performs communication processing via a network such as the Internet. Drivers 1110 are also connected to I/O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1110 as needed so that a computer program read therefrom is installed into the storage section 1108 as needed.

In particular, according to the disclosed embodiments of the present invention, the processes described in the main step diagrams above may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods illustrated in the main step diagrams. In such an embodiment, the computer program may be downloaded and installed from the network via the communication portion 1109, and/or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, the above-described functions defined in the system of the present invention are executed.

It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or operations, or can be implemented using A combination of dedicated hardware and computer instructions is implemented.

The modules involved in the embodiments of the present invention may be implemented in a software manner, and may also be implemented in a hardware manner. The described modules can also be provided in the processor, for example, it can be described as: a processor includes a determination module, a creation module and a writing module. Among them, the names of these modules do not constitute a limitation of the module itself under certain circumstances. For example, the determination module can also be described as "obtaining database monitoring information, and determining whether to Modules that need to expand the database".

As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or may exist alone without being assembled into the device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by a device, the device includes: acquiring database monitoring information, and determining according to preconfigured expansion rules and the database monitoring information. Whether the database needs to be expanded; if the database needs to be expanded, at least one new database table is established according to the current database fragmentation rules; according to the preset routing read and write rules, the original data of the database and the new The incremental data are respectively written into the corresponding library table; wherein, the library table includes the new library table.

As can be seen from the above description, after it is determined that the database needs to be expanded, a new database table is added according to the current database fragmentation rules, and data is written into the corresponding database table based on the routing read and write rules, and automatic database partitioning can be realized without downtime for deployment Table sharding and data migration; pre-configure the content contained in the expansion rules according to business requirements, and trigger the expansion process after the database monitoring information satisfies the expansion rules to realize automatic expansion; according to the database and data set in the current database sharding rules The number of tables, as well as the number of databases and data tables corresponding to the current business system, create new databases and data tables, and realize automatic database and table addition; Automatic data migration is realized by filtering and duplicating the determined database table; the routing read and write rules are implemented through the ring hash algorithm, which can ensure the balanced distribution of data to each database table; by updating the expansion rules, the expansion can be determined correctly and conveniently. Whether the subsequent database needs to be expanded again.

The above product can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by the embodiment of the present invention.

The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (12)

1. A method for expanding database, comprising:

acquiring database monitoring information, and judging whether the database needs to be expanded according to a preset expansion rule and the database monitoring information;

if the database needs to be expanded, establishing at least one new database table according to the current database fragmentation rule;

respectively writing the original data and the newly added data of the database into corresponding base tables according to a preset routing read-write rule; wherein the library table comprises the new library table.

2. The method of claim 1, wherein the database monitoring information comprises at least one of: data storage capacity, connection utilization rate and single-table data volume;

the capacity expansion rule comprises at least one of the following items: when the data storage capacity is larger than a first threshold value, the capacity of the database needs to be expanded; when the connection utilization rate is greater than a second threshold value, the capacity of the database needs to be expanded; and when the data volume of the single table is larger than a third threshold value, the capacity of the database needs to be expanded.

3. The method according to claim 1, wherein the database sharding rule sets the number of the database and the data table after capacity expansion;

the establishing of at least one new base table according to the current database fragmentation rule comprises: and newly building a database and a data table with corresponding quantity according to the quantity of the databases, the quantity of the data tables in the databases and the quantity of the expanded databases and data tables set in the current database fragmentation rule.

4. The method according to claim 1, wherein the writing the original data and the new data of the database into corresponding database tables according to a preset routing read-write rule comprises:

determining a base table to which each piece of data of the database belongs according to a preset routing read-write rule;

filtering original data in an original base table, writing the filtered data into the new base table to which the data belongs, and recording the current filtering starting time;

judging whether the original data is filtered or not, if not, determining the next filtering start time according to the current filtering start time and the filtering interval so as to continue to filter the next time until the original data is completely filtered;

and writing the newly added data into the attributive library table.

5. The method according to any one of claims 1 to 4, further comprising: and judging whether the capacity expansion is successful or not, and if the capacity expansion is successful, updating the capacity expansion rule.

6. An apparatus for expanding a database, comprising:

the judging module is used for acquiring database monitoring information and judging whether the database needs to be expanded or not according to a preset expansion rule and the database monitoring information;

the establishing module is used for establishing at least one new base table according to the current database fragmentation rule if the database needs to be expanded;

the writing module is used for respectively writing the original data and the newly added data of the database into corresponding base tables according to a preset routing read-write rule; wherein the library table comprises the new library table.

7. The apparatus of claim 6, wherein the database monitoring information comprises at least one of: data storage capacity, connection utilization rate and single-table data volume;

the capacity expansion rule comprises at least one of the following items: when the data storage capacity is larger than a first threshold value, the capacity of the database needs to be expanded; when the connection utilization rate is greater than a second threshold value, the capacity of the database needs to be expanded; and when the data volume of the single table is larger than a third threshold value, the capacity of the database needs to be expanded.

8. The apparatus according to claim 6, wherein the database sharding rule sets the number of the expanded databases and data tables;

the establishing module is further configured to: and newly building a database and a data table with corresponding quantity according to the quantity of the databases, the quantity of the data tables in the databases and the quantity of the expanded databases and data tables set in the current database fragmentation rule.

9. The apparatus of claim 6, wherein the write module is further configured to:

determining a base table to which each piece of data of the database belongs according to a preset routing read-write rule;

filtering original data in an original base table, writing the filtered data into the new base table to which the data belongs, and recording the current filtering starting time;

judging whether the original data is filtered or not, if not, determining the next filtering start time according to the current filtering start time and the filtering interval so as to continue to filter the next time until the original data is completely filtered; and

and writing the newly added data into the attributive library table.

10. The apparatus of any of claims 6 to 9, further comprising: and the updating module is used for judging whether the capacity expansion is successful or not, and updating the capacity expansion rule if the capacity expansion is successful.

11. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-5.

12. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-5.

Images