CN115460226A - A data synchronization management system, method, equipment, medium and product - Google Patents

Abstract

The embodiment of the application discloses a data synchronization management system, a method, a device, a medium and a product, wherein the system comprises: comprises at least two server groups; each server group comprises a data synchronization process and at least one service server; each business server is associated with the data synchronization process; for each server group, the service server is used for generating a data synchronization message containing service data to be synchronized when the data cross-group synchronization requirement is met, and sending the data synchronization message to the associated data synchronization process; and the data synchronization process is used for determining a target server group to be subjected to cross-group synchronization according to the received data synchronization message, and transmitting the service data to be synchronized to a target service server of the target server group through the generated synchronization data packet. The flexibility and the effectiveness of the application service data synchronization under the large-scale internet architecture are better ensured, and the data synchronization universality under the large-scale internet architecture is better reflected.

Description

A data synchronization management system, method, equipment, medium and product

technical field

The embodiments of the present application relate to the technical field of data synchronization, and in particular to a data synchronization management system, method, device, medium, and product.

Background technique

With the rapid development of the Internet era, computer storage capabilities and complex algorithms continue to develop. In recent years, the amount of data has grown exponentially. Users have higher and higher requirements for the quality of data synchronization. It is necessary to ensure the accuracy, timeliness and safety.

In the deployment of the enterprise Internet, due to the continuous increase in the amount of data, cross-regional deployment of the Internet with multiple server groups is often considered. However, under the Internet architecture of cross-regional multi-server clusters, data synchronization between server clusters often fails due to network instability or abnormal business systems. In addition, the existing data synchronization methods are often set for a certain business on the server farm, and the execution of the method is limited by the business scenario, which is not applicable.

Contents of the invention

Embodiments of the present application provide a data synchronization management system, method, device, medium, and product, which are suitable for data synchronization management of different business applications and improve the versatility of data synchronization management.

In the first aspect, the embodiment of the present invention provides a data synchronization management system, including at least two server groups;

Each server group includes at least one business server and a data synchronization process associated with each business server;

For each server farm,

The business server is used to generate a data synchronization message containing the business data to be synchronized and send it to the associated data synchronization process when there is a need for data cross-group synchronization;

The data synchronization process is used to determine the target server group to be synchronized across groups according to the received data synchronization message, and transmit the service data to be synchronized to the target service server of the target server group through the generated synchronization data packet.

In the second aspect, the embodiment of the present invention provides a data synchronization management method, which is executed by the business server and the data synchronization process in the server group included in the data synchronization management system provided in the first aspect of the embodiment of the present invention. The method includes:

For each server group, when the included business servers have data cross-group synchronization requirements, generate a data synchronization message containing the business data to be synchronized, and send it to the associated data synchronization process;

The data synchronization process determines the target server group to be synchronized across groups according to the received data synchronization message, and transmits the service data to be synchronized to the target service server of the target server group through the generated synchronization data packet.

In the third aspect, the embodiment of the present invention provides a data synchronization management device, as the execution device of the data synchronization process in the server group included in the data synchronization management system provided in the first aspect of the embodiment of the present invention, including:

one or more processors;

storage means for storing one or more programs;

One or more programs are executed by one or more processors, so that the one or more processors are used to implement the data synchronization management method provided in the second aspect of the embodiments of the present invention.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, and is characterized in that, when the program is executed by a processor, the data synchronization management method provided in the above-mentioned second aspect of the embodiment of the present invention is implemented .

In the fifth aspect, the embodiment of the present invention provides a computer program product, which is characterized in that the computer program product includes a computer program, and when the computer program is executed by a processor, the data synchronization management provided according to the above-mentioned second aspect of the embodiment of the present invention is realized method.

The embodiment of the application discloses a data synchronization management system, method, device, medium and product. The system includes: including at least two server groups, each server group includes at least one business server and a data synchronization process associated with each business server; for each server group, the business server is used when there is a need for data cross-group synchronization , generate a data synchronization message containing the business data to be synchronized, and send it to the associated data synchronization process; the data synchronization process is used to determine the target server group to be synchronized across groups according to the received data synchronization message, and pass the generated The synchronization data package transmits the service data to be synchronized to the target service server of the target server group. With the above technical solution, each business server integrated in the server group can effectively synchronize business data to other server groups to be synchronized through the data synchronization process in the server group when there is a need for cross-group synchronization of data. As a general-purpose data synchronization management system, this technical solution does not need to consider what kind of business application needs data synchronization on the business server when realizing data synchronization, and can directly realize the forwarding of data synchronization messages through the associated data synchronization process, realizing multiple Data synchronization of different business types. It better guarantees the flexibility and effectiveness of application business data synchronization under the large-scale Internet architecture, and better reflects the versatility of data synchronization under the large-scale Internet architecture.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will be easily understood from the following description.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 shows a structural block diagram of a data synchronization management system provided in Embodiment 1 of the present application;

FIG. 2 shows another structural block diagram of a data synchronization management system provided in Embodiment 1 of the present application;

FIG. 3 shows a schematic flowchart of a data synchronization management method provided in Embodiment 2 of the present application;

FIG. 4 shows another schematic flowchart of a data synchronization management method provided in Embodiment 2 of the present application;

FIG. 5 shows a schematic diagram of a hardware structure of a data synchronization management device provided in Embodiment 3 of the present application.

detailed description

In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiment of the application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the application. Obviously, the described embodiment is only It is an embodiment of a part of the application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the scope of protection of this application.

It should be noted that the terms "first" and "second" in the description and claims of the present application and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

Embodiment one

Figure 1 shows a structural block diagram of a data synchronization management system provided by Embodiment 1 of the present application. The system is suitable for data synchronization management of different business applications under the cross-regional Internet architecture. As shown in Figure 1, the data The synchronization management system includes: at least two server farms 10;

Each server group 10 includes a data synchronization process 102 and at least one business server 101; each business server 101 is associated with the data synchronization process 102;

For each server farm 10,

The business server 101 is configured to generate a data synchronization message containing the business data to be synchronized and send it to the associated data synchronization process 102 when there is a need for data cross-group synchronization;

The data synchronization process 102 is configured to determine the target server group to be synchronized across groups according to the received data synchronization message, and transfer the business data to be synchronized to the target server group through the generated synchronization data packet on the business server.

A data synchronization management system provided by an embodiment of the present invention includes at least two server groups 10, each server group 10 includes a data synchronization process 102 and at least one business server 101, and each business server 101 is associated with the data synchronization process 102 . Wherein, the server cluster (Server cluster), that is, a server cluster, gathers at least two servers to support different types of business operations. It is a large-scale Internet framework that can provide business services across regions, realize multi-region synchronization of business, and effectively avoid the disruption of the operation of clustered resources due to a server failure.

In this embodiment, the provided data synchronization management system is mainly used for receiving and processing different types of data synchronization tasks delivered from different clients in each server group. Generally, the server farm can be regarded as a distributed system architecture, and can realize global service deployment. Under this distributed system architecture, the distributed data synchronization in this embodiment can be formed by deploying server farms in different regions. management system.

It should be noted that the application scenario of the data synchronization management system can be described as: in each server group 10, each service server 101 receives service processing requirements from different service clients, and queries or receives the Regional data synchronization requirements, the data synchronization process 102 associated with each business server 101 receives the data synchronization message transmitted by each business server 101 respectively, and forwards the business data to be synchronized directly to each target business server according to the data synchronization message, For each of the cross-group synchronization requirements, the business cross-group synchronization service is directly performed. Optionally, there are at least two server groups 10 . Exemplarily, the server group 10 may be an Asian server group and a European server group, etc., and the service server 101 in the Asian server group receives data synchronization requirements across groups and transmits it to the target service server in the European server group. When one or more clients in Asia have data cross-group synchronization requirements for services such as voice calls or instant messaging services to several European clients, the data cross-group synchronization requirements are covered by different services in the Asian server group. The server 101 receives it and transmits it to certain target business servers in the European server group according to the data synchronization process 102, so as to realize the cross-cluster synchronization service of different business data.

For each server farm 10,

The service server 101 is configured to generate a data synchronization message containing the service data to be synchronized and send it to the associated data synchronization process 102 when there is a requirement for cross-group data synchronization. At least one business server 101 in the server group 10 is a server used to run business, and the business server 101 is set to respond to the business to be processed. Different business servers 101 can respond to different businesses, and can provide a service matching application. Perform processing accordingly.

Exemplarily, when a business participant has a certain demand, the client can generate a business request for this type of demand, and the business server 101 can process the business according to the business request after receiving the business request. For example, the customer When the terminal generates a synchronization request of different types of services such as instant messaging, video playback, or online conference, different service servers 101 are provided in the server group 10 to respectively process the above-mentioned different services.

The data cross-group synchronization requirement is a cross-region or cross-multi-region data synchronization request generated during the operation of the client business. When the business server 101 responds to the business of the client, the business itself has or the business server 101 queries The business to the client has cross-regional synchronization requirements (data cross-group synchronization requirements), and generates data synchronization messages containing business data to be synchronized.

Exemplarily, the service server 101 can perform service synchronization of different service types at the same time, for example, the first service server 101 can support the above-mentioned service synchronization of instant messaging, the second service server 101 can perform the above-mentioned service synchronization of video playback, and the third service server 101 can support the above-mentioned service synchronization of instant messaging. Each business server 101 can execute the business synchronization of the above-mentioned online conference. Similarly, each business server 101 in the server group 10 in the data synchronization management system can provide synchronization services for different types of businesses at the same time. It should be noted that, if the service synchronization requires cross-group synchronization, the service server 101 can also realize cross-group data synchronization of multiple service types at the same time.

Described business data to be synchronized can be business Binlog (Binary Log) data, and this business Binlog data is a kind of binary log file, is used for recording the SQL statement that the data update to Mysql or potential generation update, when data is written into database At the same time, the updated SQL statement will be written to the corresponding Binlog file at the same time. Before using Binlog, you first need to confirm whether Binlog is enabled.

The data synchronization message includes service data to be synchronized, and the service data to be synchronized is encapsulated into a format of a data synchronization message, and is sent by the service server 101 to the data synchronization process 102 . In this embodiment, the operation of generating the data synchronization message may be performed by the service server 101 after receiving a data cross-group synchronization requirement. Specifically, the business server can obtain the business data to be synchronized, and can also determine the business type of the business data to be synchronized, and can also determine which server group the business data to be synchronized to be synchronized to, thereby obtaining all The group address information that is expected to be synchronized to the server group. The service server can encapsulate the acquired data to be synchronized, the service type and the group address information expected to be synchronized to the server group, and finally form a data synchronization message.

In this embodiment, the data synchronization process 102 is configured to determine the target server group to be synchronized across groups according to the received data synchronization message, and transfer the business data to be synchronized to the server group through the generated synchronization data packet. on the target service server of the target server group. In this embodiment, one data synchronization process 102 is created in one server group 10 , that is, there is one data synchronization process 102 corresponding to one server group 10 . The data synchronization process 102 can be created on any server included in the server group 10 , and the data synchronization process can establish a communication connection with at least one service server 101 in the server group 10 .

In this embodiment, the data synchronization process 102 in the same server group 10 as the service server 101 can receive the data synchronization message, and can analyze the data synchronization message to determine that the service server 101 is to be synchronized across groups target server farm. In this embodiment, the target server group is another server group 10 in the data synchronization management system, and the target server group may also include at least one business server 101 and data created on any server in the server group 10 Synchronization process 102 .

Following the above description, after the target server group is determined, the data synchronization process 102 can generate a synchronization data packet according to the message content contained in the data synchronization message, and send the synchronization data packet to the data synchronization server on the target server group. Process 102. The target service server may be determined from multiple service servers contained in the target server group by the data synchronization process 102 in the target server group according to the received synchronization data packet. Wherein, the target service server may also be the service server 101, the two are in a relative target relationship, and the service server 101 may also exist as the target service server.

Specifically, the data synchronization process 102 on the target server group can obtain the service type in the synchronization data packet, and then select the service server 101 that matches the service type from many service servers 101 as the target service server; finally, the target server The data synchronization process 102 on the group can synchronize the service data to be synchronized in the synchronization data package to the target service server.

Exemplarily, after receiving the business service request generated by the client, the business server 101 can respond to the operation of the business service request, and if it detects that the business service has cross-group synchronization data during the operation of the business service, it can determine that it has data cross-group synchronization. Group synchronization requirements. After that, the business type of the business service, the target group identifier to be synchronized across groups, the business data to be synchronized and the synchronization configuration parameters can be obtained, and encapsulated according to the set synchronization protocol message format to form a data synchronization message; finally The generated data synchronization message is sent to the data synchronization process 102 created on the server group where it is located. After receiving the data synchronization message, the data synchronization process 102 generates a synchronization data packet, and sends the synchronization data packet from the server group according to the target group identification. The data synchronization process 102 of 10 is transferred to the target server group, so that the target business service group can transmit the business data to be synchronized in the synchronization data packet to the target business service group determined from the target server group according to the data synchronization process 102 created on it. on the server.

Wherein, a server group configuration file is stored in each of the server groups;

The server group configuration file includes: group identifiers and group addresses of each of the server groups, and service types and server addresses of service servers included in each of the server groups.

The group identifier and group address are the unique identity (Identity document, ID) and IP (Internet Protocol, Internet Protocol) address of each server group 10. Similarly, each service server address is also unique. Each server group 10 and service server 101 has its own unique identity and Internet protocol address. The group address and server address are a logical Internet address rather than addresses in the physical sense.

It should be noted that after each service server 101 receives the data synchronization request corresponding to different business types, it can obtain the business type of the business service, the target group identifier to be synchronized across groups, the business data to be synchronized and the synchronization Configure parameters, and encapsulate it according to the set synchronization protocol message format to form a data synchronization message, generate respective synchronization data packets in the data synchronization process 102, and according to the target to be cross-group synchronization corresponding to each business type The group ID is sent to its corresponding different target server groups to realize the cross-group synchronization of general data of various business types.

A data synchronization management system provided in Embodiment 1 includes at least two server groups, each server group includes at least one business server and a data synchronization process associated with each business server, and for each business server in the server group, It is used to generate a data synchronization message containing the business data to be synchronized when there is a need for data synchronization across groups, and send it to the associated data synchronization process. The data synchronization process is used to determine the data to be synchronized according to the received data synchronization message. A target server group for cross-group synchronization, and transmits the service data to be synchronized to the target service server of the target server group through the generated synchronization data packet. Each business server integrated in the server group can effectively synchronize business data to other server groups to be synchronized through the data synchronization process in the server group when there is a need for cross-group synchronization of data. As a general-purpose data synchronization management system, this technical solution does not need to consider what kind of business application requires data synchronization on the business server when realizing data synchronization. No matter which business server has a cross-group synchronization requirement during business operation, it can directly Through the generalization of the associated data synchronization process, the forwarding of the data synchronization message can be realized, and the data synchronization of various business systems can be supported at the same time. It better guarantees the flexibility and effectiveness of application business data synchronization under the large-scale Internet architecture, and better reflects the versatility of data synchronization under the large-scale Internet architecture.

As a first optional embodiment of this embodiment, on the basis of the foregoing embodiments, this first optional embodiment may optimize the service server to specifically perform the following steps:

a1) When it is detected that there is cross-group synchronization data in the operation of the business service, it is determined that there is a requirement for data cross-group synchronization.

Each server group 10 includes a data synchronization process 102 and at least one business server 101 associated with the data synchronization process 102, and the business server 101 receives business service requests from different clients.

Optionally, the business service request may be in the server group 10, using data interaction between different business servers 101 and the data synchronization process 102, that is, the target server group may be the server group 10, and the target business server may be Another service server 101 in the server group 10. Data synchronization can be realized within the same server group 10 .

Optionally, the business service request may be cross-server group, and the business server 101 in the server group 10 detects that the data in the business service operation needs to be synchronized across the group, that is, the business request for data synchronization across the server group, and determines This business has data synchronization requirements across groups.

b1) Obtain the business type of the business service, the target group identifier to be synchronized across groups, and the business data to be synchronized, and obtain synchronization configuration parameters relative to the business service.

The service type is a service type for which the client needs to synchronize data across groups, for example, it may be a service type such as message transmission, video call, or access application. The target group identifier to be synchronized across groups is the identity of the target server group that needs to perform cross-group data synchronization. Each server group 10 has an identity that belongs to the server group, and the identity is unique and definite. The synchronization configuration parameters relative to the business service may be used to configure synchronization data, for example, the sequence, attribute or size of synchronization data is processed according to preset configuration parameters.

Specifically, after the business server 101 determines the data cross-group synchronization requirement, it starts to obtain the specific information of the business to be synchronized, for example, according to the accepted business service request, it obtains the business type of the business service, the target group identifier to be synchronized across groups, and the Business data, and obtain the synchronization configuration parameters relative to the business service.

c1) Encapsulate the service type, target group identifier, service data to be synchronized and the synchronization configuration parameters according to the set synchronization protocol message format to form the data synchronization message and send it to the associated data synchronization process.

The data synchronization message includes service type, target group identifier, service data to be synchronized and synchronization configuration parameters. Wherein, the format of the synchronization protocol message forming the data synchronization message may be GRE encapsulation protocol, TCP/UDP protocol, IP protocol or LCP protocol data message, etc., and this embodiment does not limit the synchronization protocol format of the encapsulated data message. The data synchronization message integrates the above data information, and encapsulates scattered information into a data synchronization message. The service server 101 sends the encapsulated data synchronization message to the data synchronization process 102 associated with it.

A business server provided by an optional embodiment of the present invention detects that there is cross-group synchronization data in the operation of the business service, determines that it has the data cross-group synchronization requirement, and obtains the business type of the business service and the target to be cross-group synchronization The group ID and the business data to be synchronized are obtained, and the synchronization configuration parameters corresponding to the business service are obtained, and the above content is encapsulated according to the set synchronization protocol message format to form a data synchronization message and sent to the associated data synchronization process. This technical solution encapsulates the business type of the business to be synchronized, the target group identifier to be synchronized across groups, the business data to be synchronized, and the synchronization configuration parameters relative to the business service into the form of a data synchronization message. The content of the message is identified and forwarded to the target server group and the target business server, without customizing the data synchronization mechanism according to the specific business, providing a logic execution basis for generalized data synchronization under the large-scale Internet architecture, and ensuring the flexibility of application business data synchronization sex.

As a second optional embodiment of this embodiment, on the basis of the above embodiments, Fig. 2 shows another structural block diagram of a data synchronization management system provided by this embodiment, specifically, this second An optional embodiment may optimize the data synchronization process 102. As shown in FIG.

Among them, the first sub-process 1021 is used to analyze the received data synchronization message, and obtain the included service type, target group identifier, service data to be synchronized and the synchronization configuration parameters, the synchronization configuration parameters include synchronization priority level and compression settings parameters.

In the data synchronization process 102, firstly, the data synchronization message from the service server 101 is analyzed, and the analyzed service type, target group identifier, service data to be synchronized and synchronization configuration parameters are obtained. Wherein, the data synchronization message is data synchronization completed according to the standard synchronization protocol, reads the binlog log of the business data to be synchronized, scans the configured binlog storage path, parses the synchronization protocol data and extracts relevant field information. If there is a new unfinished synchronization data file, the content of the file is parsed according to the synchronization protocol. The synchronization configuration parameters include synchronization priority and compression setting parameters. Based on the synchronization priority, in the business server 101's ranking of services to be synchronized, the data synchronization business with higher priority can be faster and more accurate. Synchronize business data to the target business server. The compression setting parameters may include parameters such as whether to compress, compression format, compression method, or compression rate, and can compress the service data to be synchronized into required target service data according to the compression setting parameters. The specific compression setting parameters are not limited in this embodiment.

Specifically, the data synchronization message is transmitted synchronously according to the standard synchronization protocol. After the data synchronization message has been transmitted to the data synchronization process 102, the data synchronization message is parsed in the first sub-process 1021 to obtain the packaged data. The service type, target group identifier, service data to be synchronized, and synchronization configuration parameters of the synchronization message facilitate the effective use of the above information in subsequent processes.

Optionally, after receiving and parsing the data synchronization message, the data synchronization process 102 persists the data synchronization message, which can effectively prevent data synchronization failure caused by data loss. After the data is persistently stored, the data synchronization process 102 will feed back the data synchronization message received to the associated service server 101 that sent the data synchronization message.

The second sub-process 1022 is configured to determine a target server group to be synchronized across groups based on the target group identifier, and determine a target group address of the target server group by searching a preset server group configuration file.

Wherein, the preset server group configuration file is stored in each of the server groups 10, and the server group configuration file includes: the group identification and group address of each of the server groups 10, and the It includes the service type and server address of the service server 101.

The target group identifier of the target server group is unique and definite, and the target server group to be synchronized across groups is determined according to the unique target group identifier, and the address of the target group is searched and determined in the server group configuration file according to the target group identifier.

Specifically, the parsed data synchronization message includes the target group identifier, and the target server group for the business service can be accurately determined according to the target group identifier, and each server group 10 includes its own server group configuration file, according to the target group identifier, use the server group configuration file to find out the address of the target group, and provide a logical location basis for subsequent sub-processes.

The third sub-process 1023 is configured to form a synchronization data packet according to the service type, compression setting parameters and the service data to be synchronized after determining that the synchronization condition of the business service is currently satisfied based on the synchronization priority.

Based on the synchronization priority, it is determined whether the data to be synchronized of the current service is the first-priority synchronization service data, and if the priority of the current service to be synchronized is not the first, the data is packaged for the first-priority service to be synchronized. If the current priority of the service to be synchronized is the first, the service type of the service to be synchronized, the compression setting parameters and the service data to be synchronized are formed into a synchronization data packet.

The fourth subprocess 1024 is configured to send the synchronization data packet to the target server group according to the address of the target group.

Based on the target group address obtained from the server group configuration file, the synchronization data packet is sent from the server group 10 to the target server group according to the uniquely determined IP address.

On the basis of the foregoing embodiments, in this second optional embodiment, the third subprocess 1023 may be optimized to perform the following steps:

a2) Add the business service to the synchronization cache table according to the synchronization priority.

The synchronization cache table can simultaneously store multiple business services to be synchronized and provide static table cache. The synchronization cache table contains a plurality of services to be synchronized, and according to the synchronization priority, the services to be synchronized are sorted in order of priority from high to low, and the services to be synchronized with high priority are ranked first in the synchronization cache table One, when invoking the service data to be synchronized, the service data to be synchronized with the first priority is called to the synchronization process first.

b2) When it is detected that the business service is called out from the synchronization cache table, it is determined that the synchronization condition of the business service is met.

The synchronization condition refers to determining that the business service called out from the synchronization cache table has a cross-group synchronization requirement and ranks first in real time in the priority ranking in the synchronization cache table. The synchronization condition of the business service can be met, and the next step of the synchronization process can be carried out.

c2) If the compression setting parameter is data compression, compress the service data to be synchronized to obtain target service data; otherwise, use the data to be synchronized as target service data.

Determine whether to compress the business data to be synchronized in the corresponding compression setting parameters, if data compression is required, perform corresponding data compression processing on the business data to be synchronized according to other compression parameters in the preset compression setting parameters, and obtain the target business data.

Specifically, if the compression setting parameter is set to compress data, the service data to be synchronized can continue to be compressed according to other compression parameters in the compression setting parameter, such as compression format, compression method or compression rate, etc. After the compression setting parameter compresses the service data to be synchronized, the target service data is obtained according to the compression result; if the compression setting parameter is set to not compress the service data to be synchronized, then the service data to be synchronized itself is the target service data.

d2) encapsulating the service type and the target service data according to the set format to form a synchronous data packet.

The set encapsulation format is the same as the encapsulation format in step c1 in the above-mentioned embodiment, and will not be described in detail here in this embodiment. The operation of generating the synchronization data packet may be performed by the data synchronization process 102 after determining that the synchronization condition is satisfied. Specifically, the first subprocess 1021 obtains and parses the data synchronization message, obtains the included service type, target group identifier, service data to be synchronized, and the synchronization configuration parameters, and determines that the synchronization condition is met, that is, the priority of the service to be synchronized When it is the first place, the business type obtained after parsing the data synchronization message and the processed target business data are encapsulated into a synchronization data packet according to the set format.

The target service data may be compressed service data to be synchronized, or the service data to be synchronized itself, and the target service data and service type are encapsulated in a set format to form a synchronization data packet.

A data synchronization process provided by an optional embodiment of the present invention includes a first sub-process for parsing a received data synchronization message to obtain the included service type, target group identifier, service data to be synchronized and synchronization configuration parameters ; The second sub-process is used to determine the target server group to be synchronized across groups based on the target group identifier, and determine the target group address of the target server group by searching the preset server group configuration file; the third sub-process is used to synchronize based on After the priority is determined to meet the synchronization conditions of the business service, a synchronization data packet is formed according to the business type, compression setting parameters and business data to be synchronized; the fourth sub-process is used to send the synchronization data packet to the target server according to the target group address group. The technical scheme of the invention determines the correct service data to be synchronized and generates a synchronization data packet, and sends the synchronization data packet according to the identification and address of the target server group corresponding to the service. The synchronization data packet, which is different from the prior art, can be directly forwarded to the target server group through the target group identifier and address, effectively confirming the accuracy of the target business data and the efficiency of data cross-group transmission.

As a third optional embodiment of the first embodiment, on the basis of the above embodiments, as shown in FIG. 2 , the data synchronization process 102 further includes: a fifth sub-process 1025 and a sixth sub-process 1026 .

The fifth sub-process 1025 is used to persistently save the business data to be synchronized obtained from the data synchronization message 102, and feed back the message response result to the business server 101;

The persistent saving of the service data to be synchronized obtained from the data synchronization message can effectively prevent data loss during the synchronization process, that is, save the data in a storage device (such as a disk) that can be stored permanently. The main application of persistence is to store objects in memory in a database, or in a disk file, in an XML data file, and so on. Persistence is the mechanism for converting program data between persistent and transient states. After the service data to be synchronized has been synchronized to the target service server, the persistently stored service data to be synchronized can be deleted.

Specifically, in the data synchronization process 102, the first sub-process 1021 parses the data synchronization message and obtains the service data to be synchronized in the synchronization data packet text, in order to prevent the business server 101 from breaking down or the server cluster from occurring during the data synchronization process. 10. If a failure occurs, resulting in data loss and unable to complete the data synchronization service, persist the business data to be synchronized, and feed back the response result of the data synchronization message in the data synchronization process 102 to the business server 101, and complete A small loop for data synchronization.

The sixth sub-process 1026 is configured to delete the persistently stored service data to be synchronized after sending the synchronization data packet to the target server group, and send a message to the service server after receiving the synchronization completion message fed back by the target server group. 101 Feedback the data synchronization result.

Specifically, after the synchronization data packet is sent to the target server group, the persistently stored service data to be synchronized is deleted. This process occurs in the server group 10 at the end of sending the synchronization data packet, and the data synchronization process in the server group The fifth subprocess 1025 of 102 stores the business data to be synchronized persistently. In the sixth subprocess 1026, the business data to be synchronized has been successfully sent to the target server group, and the data will not be stored in the data synchronization subprocess. In case of data loss, it can be deleted from the original server group 10 . At the same time, after receiving the completed data synchronization message fed back by the target server group, the data synchronization result is fed back to the original service server 101 to complete the message cycle in the cluster and determine that the synchronization has been completed.

A data synchronization process provided by an optional embodiment of the present invention includes a fifth subprocess for persistently storing the service data to be synchronized obtained from the data synchronization message, and feeding back a message response to the service server Result; the sixth sub-process is used to delete the persistently stored business data to be synchronized after sending the synchronization data packet to the target server group, and after receiving the synchronization completion message fed back by the target server group, send a message to the target server group The above-mentioned service server feeds back the data synchronization result. The technical scheme of the present invention saves the business data information to be synchronized when advancing the data synchronization process, effectively prevents data loss when the synchronization is not completed, and clears the business data to be synchronized and feeds back the synchronization completion message after the data synchronization is completed in stages, reducing useless messages Occupies memory, providing channel interference-free basis for subsequent data synchronization of other services. The security of data synchronization is effectively improved, and the efficiency of the data synchronization management system is reflected.

As a fourth optional embodiment of the first embodiment, on the basis of the foregoing embodiments, as shown in FIG. 2 , the data synchronization process 102 further includes a seventh subprocess 1027 .

The seventh sub-process 1027 is configured to receive a synchronization data packet sent by other server groups 10, determine a target service server according to the synchronization data packet, and synchronize the target service data in the synchronization data packet to the target server.

The synchronization data packet includes service type and target service data, and the matching target service server is determined according to the service type.

On the basis of the above optimization, the seventh subprocess 1027 can specifically be used to perform the following steps:

a3) Receive and analyze the synchronization data packet sent by any other server group 10 to obtain the service type and target service data.

When the server group 10 is used as the target server group, it receives the synchronization data packet sent from any other server group. After the synchronization data packet is stored persistently and the data synchronization result is fed back to the server group 10 that sent the synchronization data packet, the synchronization data packet is analyzed to obtain the service type and target service data.

b3) Search the preset server group configuration file, determine that the service server corresponding to the service type is recorded as the target service server, and obtain the target server address of the target service server.

Each server group 10 is configured with a preset server group configuration file, and the server group configuration file includes: the group identification and group address of each of the server groups 10, and the service servers 101 contained in each of the server groups 10 business type and server address. Search for a target service server matching the service type according to the association relationship between the service type and the server address in the server group configuration file, and obtain the address of the target service server.

c3) Sending the target service data to the target service server according to the address of the target server, so that the target service data is synchronized to the target service server.

The address of the target service server is unique, and the target service data is sent to the target service server according to the address of the target service server that matches the service type to complete data cross-group synchronization. After the target business server receives the target business data, it feeds back the data synchronization result to the server group 10 where it is located, and after the server group 10 receives the feedback result, it sends the feedback result to the server group 10 that initiates the data synchronization service After receiving the data synchronization result feedback, the server group 10 that initiates the data synchronization service sends the feedback result to the service server 101 corresponding to the service.

A data synchronization process provided by an optional embodiment of the present invention includes a seventh sub-process for receiving synchronization data packets sent by other server groups, determining the target service server according to the synchronization data packets, and synchronizing the data in the synchronization data packets The target service data is synchronized to the target server. This technical solution accurately queries the target business server according to the business type and completes the cross-group synchronization of data, without customizing the data synchronization mechanism according to the business type, and realizes the versatility, flexibility and efficiency of data synchronization.

In addition, this embodiment also optimizes that the seventh subprocess can be used to perform the following steps:

The target service data in the received synchronization data packet is persistently saved to prevent the target data service from being lost and unable to be retrieved if it is not synchronized to the target service server or during the synchronization process. Improved the security of the data synchronization system.

When a synchronization completion message fed back by the target service server relative to the target service data is received, the persistently stored target service data is deleted, and a synchronization completion message is fed back to the server group that sent the synchronization data packet. It effectively reduces the memory usage in the data synchronization system, provides sufficient synchronization space for subsequent business synchronization, and improves the synchronization efficiency of the data synchronization system.

Embodiment two

Figure 3 shows a schematic flow diagram of a data synchronization management method provided in Embodiment 2 of the present application. This method is applicable to the situation of performing data synchronization management on different business applications under a cross-regional Internet architecture. The provided data synchronization management system includes business servers and data synchronization processes in the server group to execute.

As shown in Figure 3, a data synchronization management method provided in Embodiment 2 of the present application may specifically include the following steps:

Step 301, for each server group, when the included business servers meet the data cross-group synchronization requirements, generate a data synchronization message containing the business data to be synchronized, and send it to the associated data synchronization process;

The server group is a server cluster, which gathers at least two servers to support different types of business operations. It is a large-scale Internet framework that can provide business services across regions, realize multi-region synchronization of business, and effectively avoid the disruption of the operation of clustered resources due to a server failure.

The service server is configured to generate a data synchronization message containing the service data to be synchronized and send it to an associated data synchronization process when there is a requirement for data synchronization across groups. At least one business server in the server group is a server for running business. The business server is set to respond to the business to be processed. Different business servers can respond to different businesses and provide a corresponding execution process for business matching.

Exemplarily, when a business participant has a certain demand, the client can generate a business request for this type of demand, and the business server can process the business according to the business request after receiving the business request. For example, the client When requests for different types of services such as instant messaging, video playback, or online conferences are generated, different service servers are provided in the server group to process the above-mentioned different services respectively.

The data cross-group synchronization requirement is a cross-region or cross-multi-region data synchronization request generated during the operation of the client business. When the business server responds to the client, the business itself has or the business server queries the client. The terminal business has cross-regional synchronization requirements (data cross-group synchronization requirements), and generates data synchronization messages containing business data to be synchronized. The business data to be synchronized can be business Binlog data, and the business Binlog data is a binary log file used to record data updates to Mysql or potentially updated SQL statements. When data is written into the database, it will also At the same time, write the updated SQL statement into the corresponding Binlog file. Before using Binlog, you need to confirm whether Binlog is enabled.

The data synchronization message includes service data to be synchronized, and the service data to be synchronized is encapsulated into a format of a data synchronization message, which is sent to the data synchronization process by the service server. In this embodiment, the operation of generating the data synchronization message may be performed by the service server after receiving a data cross-group synchronization requirement. Specifically, the business server can obtain the business data to be synchronized, and can also determine the business type of the business data to be synchronized, and can also determine which server group the business data to be synchronized to be synchronized to, thereby obtaining all The group address information that is expected to be synchronized to the server group. The service server can encapsulate the acquired data to be synchronized, the service type and the group address information expected to be synchronized to the server group, and finally form a data synchronization message. And send the data synchronization message to the data synchronization process.

Step 302, the data synchronization process determines the target server group to be synchronized across groups according to the received data synchronization message, and transmits the business data to be synchronized to the target server group through the generated synchronization data packet on the target business server.

In this embodiment, one data synchronization process is created in one server group, that is, there is one data synchronization process corresponding to one server group. The data synchronization process can be created on any server included in the server group, and the data synchronization process can establish a communication connection with at least one service server in the server group.

In this embodiment, the data synchronization process in the same server group as the business server can receive the data synchronization message, and can analyze the data synchronization message to determine the target server group for the business server to be synchronized across groups . In this embodiment, the target server group is another server group in the data synchronization management system, and the target server group may also include at least one business server and a data synchronization process created on any server in the server group.

Following the above description, after the target server group is determined, the data synchronization process can generate a synchronization data packet according to the message content contained in the data synchronization message, and send the synchronization data packet to the data synchronization process on the target server group . The target service server can be determined from multiple service servers contained in the target server group by the data synchronization process in the target server group according to the received synchronization data packet.

Specifically, the data synchronization process on the target server group can obtain the service type in the synchronization data packet, and then select the service server matching the service type from many service servers as the target service server; finally, the target server group The data synchronization process can synchronize the service data to be synchronized in the synchronization data package to the target service server.

The embodiment of the present application discloses a data synchronization management method. For each server group, when the included business server has the requirement of cross-group data synchronization, it generates a data synchronization message containing the business data to be synchronized and sends it to the associated A data synchronization process, wherein the data synchronization process determines the target server group to be synchronized across groups according to the received data synchronization message, and transmits the business data to be synchronized to the target server group through the generated synchronization data packet on the target business server. With the above technical solution, each business server integrated in the server group can effectively synchronize business data to other server groups to be synchronized through the data synchronization process in the server group when there is a need for cross-group synchronization of data. This technical solution does not need to consider what kind of business application needs data synchronization on the business server when realizing data synchronization. As long as there is a need for data synchronization, it can directly realize the forwarding of data synchronization messages through the generalization of the associated data synchronization process. It better guarantees the flexibility and effectiveness of application business data synchronization under the large-scale Internet architecture, and better reflects the versatility of data synchronization under the large-scale Internet architecture.

In order to facilitate a better understanding of the data synchronization management system provided in the above embodiment, an application example is given below in this embodiment to illustrate the specific implementation of the data synchronization management method. FIG. 4 shows an example flow chart of a data synchronization management method provided by an embodiment of the present application.

As shown in Figure 4, taking the data synchronization between two server groups under the data synchronization management system as an example, each server group includes a data synchronization process and at least one business server. In this example, one of the server groups is used as A source server group with cross-group synchronization data, and another server group is used as a target server group for synchronizing business data on the source server group.

Specifically, the implementation of the data synchronization management method includes the following steps:

S1. Encapsulate the service type, target group identifier, service data to be synchronized, and synchronization configuration parameters into a data synchronization message.

The data synchronization system includes at least two server groups, each server group includes at least one business server and a data synchronization process associated with each of the business servers, and for each business server in the server group, it is detected that the business service is running When there is cross-group synchronization data in , determine that there is a requirement for data cross-group synchronization, and obtain the business type of the business service, the target group identifier to be synchronized across groups, and the business data to be synchronized, and obtain the synchronization relative to the business service Configuring parameters, encapsulating the service type, target group identifier, service data to be synchronized, and the synchronization configuration parameters according to the set synchronization protocol message format to form the data synchronization message. The synchronization configuration parameters include synchronization priority and compression setting parameters.

S2. Sending a data synchronization message.

The service server sends the encapsulated data synchronization message to the associated data synchronization process.

S3. Compress and persist the data synchronization message.

In the data synchronization process, the data synchronization message is compressed according to the compression setting parameters, and the data synchronization message is persistently stored in the data synchronization process.

S4. Feedback the message response result to the service server.

After the data synchronization process receives and persists the data synchronization message, it feeds back the message response result to the service server that sent the data synchronization message.

S5. Parse the data synchronization message to obtain the included service type, target group identifier, service data to be synchronized and synchronization configuration parameters.

S6. Determine a target server group to be synchronized across groups based on the target group identifier.

A target server group to be synchronized across groups is determined based on the target group identifier, and a target group address of the target server group is determined by searching a preset server group configuration file. The server group configuration file includes: group identifiers and group addresses of each of the server groups, and service types and server addresses of service servers included in each of the server groups.

S7. Form a synchronization data packet according to the service type, compression setting parameters and service data to be synchronized, and send it to the target service server group.

When it is detected that the business service is called out from the synchronization cache table, after determining that the synchronization condition of the business service is currently met based on the synchronization priority, according to the business type, compression setting parameters and the business data to be synchronized , forming a synchronization data packet; wherein, if the compression setting parameter is to perform data compression, then perform compression processing on the service data to be synchronized to obtain target service data, otherwise, use the data to be synchronized as target service data; according to The synchronization priority adds the business service to the synchronization cache table; encapsulates the business type and the target business data according to the set format to form a synchronization data packet. According to the address of the target group, the synchronization data packet is sent to the target server group.

Optionally, persistently save the business data to be synchronized obtained from the data synchronization message, and feed back the message response result to the business server; after sending the synchronization data packet to the target server group, delete the persistent storage The service data to be synchronized, and after receiving the synchronization completion message fed back by the target server group, feed back the data synchronization result to the service server.

S8. Persistently save the synchronization data packet.

When the server group exists as the target server group, it receives the synchronization data packets sent by other server groups, and persists the synchronization data packets.

S9. Feedback a synchronization completion message to the server group sending the synchronization data packet.

S10. Parse the synchronization data packet and obtain service type and target service data.

Analyze the synchronization data packet received by the server group as the target server group to obtain the business type and target business data.

S11. Determine the target service server and obtain the address of the target server.

Searching for a preset server group configuration file, determining that the business server corresponding to the business type is recorded as the target business server, and obtaining the target server address of the target business server;

S12. Send the target service data to the target service server.

According to the address of the target server, the target service data is sent to the target service server, so that the target service data is synchronized to the target service server.

S13. A data synchronization completion message is fed back.

When a synchronization completion message fed back by the target service server relative to the target service data is received, the persistently stored target service data is deleted, and a synchronization completion message is fed back to the server group that sent the synchronization data packet.

A data synchronization management method provided in Embodiment 2 of the present application encapsulates the service type, target group identifier, service data to be synchronized, and synchronization configuration parameters into a data synchronization message and sends the data synchronization message to the data synchronization process. During the data synchronization process, the data synchronization message is compressed and persisted, the message response result is fed back to the business server, the data synchronization message is parsed, and the included business type, target group identifier, business data to be synchronized and synchronization configuration parameters are obtained, based on The target group identifier determines the target server group to be synchronized across groups, forms a synchronization data packet according to the business type, compression setting parameters and business data to be synchronized, and sends it to the target business server group, and persists the synchronization data packet in the target server group Save, feed back the synchronization completion message to the server group that sent the synchronization data packet, analyze the synchronization data packet and obtain the business type and target business data, determine the target business server and obtain the target server address, send the target business data to the target business server, and finally send The service server of the data synchronization message feeds back a data synchronization completion message. In this technical solution, each business server integrated in the server group can effectively synchronize business data to other server groups to be synchronized through the data synchronization process in the server group when there is a requirement for cross-group synchronization of data. This technical solution does not need to consider what kind of business application needs data synchronization on the business server when realizing data synchronization. As long as there is a need for data synchronization, it can directly realize the forwarding of data synchronization messages through the generalization of the associated data synchronization process. It better guarantees the flexibility and effectiveness of application business data synchronization under the large-scale Internet architecture, and better reflects the versatility of data synchronization under the large-scale Internet architecture.

Embodiment Three

FIG. 5 shows a schematic diagram of a hardware structure of a data synchronization management device provided in Embodiment 3 of the present application. An embodiment of the present application provides a data synchronization management device.

As shown in Figure 5, the data synchronization management device 50 includes at least one processor 51, and a memory connected to the at least one processor 51 in communication, such as a read-only memory (ROM) 52, a random access memory (RAM) 53, etc., wherein, The memory stores a computer program executable by at least one processor, and the processor 51 can execute the computer program according to the computer program stored in the read-only memory (ROM) 52 or loaded from the storage unit 58 into the random access memory (RAM) 53 , to perform various appropriate actions and processing. In the RAM 53, various programs and data necessary for the operation of the data synchronization management device 50 can also be stored. The processor 51 , ROM 52 and RAM 53 are connected to each other through a bus 54 . An input/output (I/O) interface 55 is also connected to the bus 54 .

Multiple components in the data synchronization management device 50 are connected to the I/O interface 55, including: an input unit 56, such as a keyboard, a mouse, etc.; an output unit 57, such as various types of displays, speakers, etc.; a storage unit 58, such as a magnetic disk , CD, etc.; and communication unit 59, such as network card, modem, wireless communication transceiver, etc. The communication unit 59 allows the data synchronization management device 50 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the data synchronization management method provided in the embodiment of the present application is implemented.

An embodiment of the present application further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the data synchronization management method provided in the embodiment of the present application is implemented.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the various steps described in this application may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution of this application can be achieved, there is no limitation herein.

The above specific implementation methods are not intended to limit the protection scope of the present application. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (13)

1. A data synchronization management system, characterized in that it comprises at least two server groups; Each of the server groups includes a data synchronization process and at least one business server; each of the business servers is associated with the data synchronization process; For each server farm, The business server is configured to generate a data synchronization message containing the business data to be synchronized and send it to the associated data synchronization process when the data cross-group synchronization requirement is met; The data synchronization process is used to determine the target server group to be synchronized across groups according to the received data synchronization message, and transmit the service data to be synchronized to the target server group through the generated synchronization data packet on the target business server. 2. The system according to claim 1, wherein the service server is specifically used for: When it is detected that there is cross-group synchronization data in the operation of business services, it is determined that there is a need for data cross-group synchronization; Obtain the business type of the business service, the target group identifier to be synchronized across groups, and the business data to be synchronized, and obtain synchronization configuration parameters relative to the business service; Encapsulate the service type, target group identifier, service data to be synchronized and the synchronization configuration parameters according to the set synchronization protocol message format to form the data synchronization message and send it to the associated data synchronization process. 3. The system according to claim 1, wherein the data synchronization process comprises: The first sub-process is used to analyze the received data synchronization message, and obtain the included service type, target group identifier, service data to be synchronized and the synchronization configuration parameters, which include synchronization priority and compression Setting parameters; The second sub-process is used to determine the target server group to be synchronized across groups based on the target group identifier, and determine the target group address of the target server group by searching a preset server group configuration file; The third sub-process is used to form a synchronization data packet according to the service type, compression setting parameters and the service data to be synchronized after determining that the synchronization condition of the business service is currently satisfied based on the synchronization priority; The fourth sub-process is configured to send the synchronization data packet to the target server group according to the target group address. 4. The system according to claim 3, wherein the third subprocess is specifically used for: Add the business service to the synchronization cache table according to the synchronization priority; When it is detected that the business service is called out from the synchronization cache table, it is determined that the synchronization condition of the business service is met; If the compression setting parameter is to perform data compression, perform compression processing on the service data to be synchronized to obtain target service data, otherwise, use the data to be synchronized as target service data; Encapsulating the service type and the target service data according to the set format to form a synchronous data packet. 5. The system according to claim 1, wherein the data synchronization process further comprises: The fifth sub-process is used for persistently storing the service data to be synchronized obtained from the data synchronization message, and feeding back a message response result to the service server; The sixth sub-process is used to delete the persistently stored service data to be synchronized after sending the synchronization data packet to the target server group, and send the service data to the service group after receiving the synchronization completion message fed back by the target server group. The server feeds back the data synchronization result. 6. The system according to claim 1, wherein when the server group where the data synchronization process is located is used as the target server group, the data synchronization process further includes: The seventh sub-process is used to receive a synchronization data packet sent by other server groups, determine a target service server according to the synchronization data packet, and synchronize the target service data in the synchronization data packet to the target server. 7. The system according to claim 6, wherein the seventh sub-process is specifically used for: Receive and analyze the synchronization data packet sent by any other server group to obtain the business type and target business data; Searching for a preset server group configuration file, determining that the business server corresponding to the business type is recorded as the target business server, and obtaining the target server address of the target business server; According to the address of the target server, the target service data is sent to the target service server, so that the target service data is synchronized to the target service server. 8. The system according to claim 6, wherein the seventh subprocess is also used for: Persistently save the target business data in the received synchronization data packet; When a synchronization completion message fed back by the target service server relative to the target service data is received, the persistently stored target service data is deleted, and a synchronization completion message is fed back to the server group that sent the synchronization data packet. 9. The system according to any one of claims 1-8, wherein a server group configuration file is stored in each of the server groups; The server group configuration file includes: group identifiers and group addresses of each of the server groups, and service types and server addresses of service servers included in each of the server groups. 10. A data synchronization management method, characterized in that it is executed by the business server and the data synchronization process in the server group included in the data synchronization management system according to any one of claims 1-9, the method comprising: For each server group, when the included business servers have data cross-group synchronization requirements, generate a data synchronization message containing the business data to be synchronized, and send it to the associated data synchronization process; The data synchronization process determines the target server group to be synchronized across groups according to the received data synchronization message, and transmits the service data to be synchronized to the target service server of the target server group through the generated synchronization data packet superior. 11. An electronic device, characterized in that, as a business server in the data synchronization management system according to any one of claims 1-9 and/or an execution device created with a data synchronization process, comprising: one or more processors; storage means for storing one or more programs; The one or more programs are executed by the one or more processors, so that the one or more processors are used to execute the data synchronization management method described in claim 10 . 12. A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, the data synchronization management method according to claim 10 is realized. 13. A computer program product, characterized in that the computer program product comprises a computer program, and when the computer program is executed by a processor, the data synchronization management method according to claim 10 is realized.

Images