CN111835855A - A data synchronization method and system - Google Patents

Abstract

The present invention discloses a data synchronization method and system. The present invention includes S1, establishing a data synchronization control table at the server, wherein the data synchronization control table records the data update time stamp of the latest data update performed by each interface, and When the server updates data, the data synchronization control table updates the corresponding data update timestamp; S2, the server provides a polling interface to the client, and the parameters of this interface are the interface of the client and the latest synchronization data of each interface. Data synchronization timestamp. The function is to compare the data synchronization timestamp of the client's interface with the corresponding data update timestamp in the data synchronization control table. If there is inconsistency, return the inconsistent interface to the client. Compared with the original situation that each interface needs to perform polling request, the present invention greatly reduces the invalid calls of the client, saves the resources of the server, reduces the pressure of the database, and promotes the service to be more stable.

Description

A data synchronization method and system

technical field

The invention relates to the technical field of computers, in particular to a data synchronization method and system.

Background technique

In the construction of network applications, the client needs to constantly communicate with the server to maintain the synchronization of the information between the two parties. Usually, this persistent communication is carried out without refreshing the page, and the "server push" scheme is used to push messages to the client. The most commonly used is short polling. The basic idea of short polling is that the client sends http requests to the server at regular intervals, and the server responds directly after receiving the request, regardless of whether there is data update. The instant communication implemented in this way is essentially a process in which the client sends a request and the server accepts the request. By allowing the client to make continuous requests, the client can simulate real-time data changes received from the server. The advantage of this method is that it is relatively simple, easy to understand, and has no technical difficulties to implement, but the disadvantage is obvious. This method seriously wastes the resources of the server and the client due to the continuous establishment of http connections. Especially on the client side, specifically, if there are relatively large people in the short polling-based application at the same time, then each user's client side will frantically send http requests to the server side without interruption, The more people, the more pressure on the server side, which is very unreasonable.

For example, whether it is a large supermarket chain or offline stores, the cloud POS cash register system is generally used for invoicing management. In order to be able to use normally even when the network is disconnected, such systems will store product data, inventory data, user data (such as new Add cashiers, etc.), institutional parameters, category information, activities, etc. are downloaded to the local client. After the above information is updated in the background, the client cannot be actively notified. If the client wants to get the latest data, it needs to use the polling method. For example, it sends a request to the server every 10 minutes and uses the incremental download method to obtain the changed data. The main logic is that when the data is modified in the background, the corresponding timestamp will be updated at the same time. The client initiates the request with the local timestamp. The server receives the request and compares the two timestamps, and only retrieves data from the database that is greater than the local timestamp. . In fact, most of the data is rarely changed. For example, user data will only be changed when a cashier is added or user information is modified. Product data will only be changed when added, modified, or deleted. The time of user background operations is also relatively concentrated. As a result, invalid requests are being sent all the time, which undoubtedly puts pressure on the database and server-side applications, wastes resources, and in extreme cases, causes the database and application to go down and services become unavailable. For example, in a cloud POS service provider, there are about 500,000 retailers nationwide, and the client has 20 regular download interfaces to send requests to the server every ten minutes, and the online time of each retailer is calculated as 10 hours per day. , 600 million requests a day in total. The main function of the client is sales. Sales mainly involve changes in the business only inventory. After sales, the inventory data will be changed. The frequency of changes is proportional to the sales of products. Therefore, only a small part of these 600 million requests are valid calls, that is, the background data corresponding to a small part of the interface will change, and the changed data will be returned.

SUMMARY OF THE INVENTION

The purpose of the present invention is to address the above deficiencies, to provide a method and system for data synchronization based on the above short polling method, which reduces the number of data interface requests and reduces server pressure.

The technical scheme adopted in the present invention is:

A data synchronization method, the method comprises the following steps:

S1, establish a data synchronization control table on the server, the data synchronization control table records the data update timestamp of the latest data update performed by each interface, and when the server updates data, the data synchronization control table updates the corresponding data update timestamp;

S2. The server provides a polling interface to the client. The parameters of this interface are the interface of the client and the data synchronization timestamp of the latest synchronization data of each interface. The function is the data synchronization timestamp of the client interface and the data synchronization control table. When there is inconsistency, the inconsistent interface is returned to the client, and the client calls the returned interface to request data synchronization from the server.

Specifically, the data synchronization control table of the present invention contains data columns as user code, interface ID and data update timestamp of the latest data update. Interface ID. When the server accepts manual data updates, the data update timestamp of the docking interface ID in the data synchronization control table is changed to the time when the data is updated.

As a further optimization of the method of the present invention, the present invention updates the data update timestamps of different interfaces and other data-related data tables by establishing a message queue when the server updates data.

As a further optimization of the method of the present invention, in step S2 of the present invention, the data synchronization time stamp of the local interface of the server client is compared with the data update time stamp in the data synchronization control table, and the data synchronization time stamp and the data update time are compared. Stamp inconsistent interface IDs back to the client.

As a further optimization to the method of the present invention, in step S2 of the present invention, the process of requesting data synchronization from the server by the interface that the client calls and returns includes:

S201, the client invokes the interface ID returned by the server and requests synchronization data from the server;

S202, the server returns the requested data to the corresponding interface, the corresponding interface data is updated, and the data synchronization timestamp of the interface is updated.

The present invention also provides a data synchronization system, including a server and a client, and the server includes a data synchronization control table, a comparison module and a data transmission module;

Wherein, the data synchronization control table is used to record the service interface of the server and the data update timestamp of the latest data update of each interface;

The comparison module is used to compare the data synchronization timestamp of the latest synchronization data of the client interface with the corresponding data update timestamp in the data synchronization control table after accepting the corresponding request of the client, and output the two timestamps to the client that are inconsistent. Interface;

The data sending module is used for the interface data synchronization request of the interface client, and sends corresponding update data to the interface.

As a further optimization of the system of the present invention, the server of the present invention further includes a data update module. When the server performs data update, and the update of the same data involves the update of the data update timestamps of multiple interfaces, the data update module passes Establish a message queue to update the timestamps of different interfaces.

The present invention also provides another data synchronization system, including a client and a server, the client includes an interface update screening module and an interface data request module, the server is provided with a data synchronization control table, the data The synchronization control table is used to record the timestamp of the latest updated data of the service interface;

Wherein: the interface update screening module is used to send a polling request to the server, and receive an interface in which the data synchronization timestamp of the client interface returned by the server is inconsistent with the corresponding data update timestamp in the data synchronization control table in the server ;

The interface data request module calls the interface received by the above-mentioned interface update screening module and requests synchronization data from the server, and receives the data returned by the server to update the original data.

The present invention has the following advantages:

1. The method of the present invention adds a data table that records the data update time point of each interface on the server side, and provides the client with a polling interface, and the client first compares the data synchronization time of the local interface through this interface. The stamp is compared with the data update timestamp in the data table. When it is inconsistent, the interface that is inconsistent with the client is returned. The client calls these interfaces to request data synchronization from the server. Compared with the original polling request for each interface, it is greatly Reduce the invalid calls of the client, save the resources of the server, reduce the pressure on the database, and make the service more stable;

2. When the data is updated, the server of the present invention updates the timestamp of the data table by means of message queue, which can reduce the complexity of the system, reduce the possibility of errors, avoid the occurrence of table lock, and the asynchronous data of the message queue. The way of updating has no effect on the method of the present invention.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

Below in conjunction with accompanying drawing, the present invention is further described:

Fig. 1 is the format schematic diagram of the data synchronization control table of the present invention;

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Hereinafter, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

It should be understood that, in the description of the embodiments of the present invention, words such as "first" and "second" are only used for the purpose of distinguishing the description, and should not be understood as indicating or implying relative importance, nor should it be understood as indicating or implied order. In the embodiments of the present invention, "a plurality" refers to two or more.

In this embodiment of the present invention, "and/or" is only an association relationship describing associated objects, indicating that three relationships may exist, for example, A and/or B may indicate that A exists alone, B exists alone, There are three cases A and B at the same time. In addition, the character "/" in this text generally indicates that the contextual object is an "or" relationship.

This embodiment provides a data synchronization method, and the method includes the following steps:

S1, establish a data synchronization control table at the server, the data synchronization control table contains data columns as user code, interface ID and the data update timestamp of the latest data update, as shown in Figure 1, USER_ID is the user code, API_ID is the interface ID, TIME_STAMP is the timestamp, the user code is used to correspond to the client, and the interface ID is the interface ID that is connected to each data request synchronization of the client. When the server accepts manual data updates, the data update time of the interface ID in the data synchronization control table The stamp is changed to the time point when the data is updated. When the server updates the data, the data synchronization control table updates the corresponding data update timestamp;

S2. The server provides a polling interface to the client. The parameters of this interface are the interface ID of the client and the data synchronization timestamp of the latest synchronization data of each interface ID. The function is the data synchronization timestamp and data of the interface ID of the client. Compare the corresponding data update timestamps in the synchronization control table. If there is inconsistency, it means that there is data in the interface that needs to be synchronized, and the inconsistent interface ID is returned to the client. The client calls the returned interface and requests data synchronization from the server. When the comparison of all interfaces is the same, it means that there is no data synchronization, and the server returns NULL to the client;

The process that the client calls the returned interface to request data synchronization from the server includes:

S201, the client invokes the interface ID returned by the server and requests synchronization data from the server;

S202, the server returns the requested data to the corresponding interface, the corresponding interface data is updated, and the data synchronization timestamp of the interface is updated.

As a preferred embodiment, in step S1, when the server performs data update, and the update of the same data involves the data update timestamps of multiple interfaces and needs to be updated, the data update time of different interfaces is updated by establishing a message queue. Click to update. When data is updated, the change of the same data will involve the change of multiple related data rows. For example, after the product information is modified in the background, the data update timestamp of the product and the data update timestamp of the total product table need to be updated. When updating the database directly, lock waiting will occur when multiple services update a record in the database at the same time, and batch updates may also cause deadlocks. Using message queues to asynchronously modify timestamps in multiple data synchronization control tables can reduce the complexity of the system. , Reduce errors and avoid the occurrence of lock table. Although the use of message queues will cause data to be updated untimely, this method uses short polling between the client and the server to update client data, so the server modifies asynchronously. The data update time stamp has no effect on the data synchronization of the present invention.

Based on the above data synchronization method, the present invention provides an embodiment of a data synchronization system, which is described from the perspective of a server, including a server and a client, and the server includes a data update module, a data synchronization control table, and a comparison module for data transmission modules and data update modules;

Wherein, the data synchronization control table is used to record the service interface of the server and the data update timestamp of the latest data update of each interface;

The comparison module is used to compare the data synchronization timestamp of the latest synchronization data of the client interface with the corresponding data update timestamp in the data synchronization control table after accepting the corresponding request of the client, and output the two timestamps to the client that are inconsistent. Interface;

The data sending module is used for the interface request of the interface client, and sends corresponding update data to the interface;

The data update module connects the time stamp update and the data update element in the data synchronization control table by means of a message queue, and after the data update element performs data update, the message queue is used to asynchronously modify the corresponding time stamp in the data synchronization control table. .

The embodiment of the present invention also provides another data synchronization system, which is described from the perspective of the client, including a client and a server, the client includes an interface update screening module and an interface data request module, and the server is provided with data synchronization a control table, the data synchronization control table is used to record the timestamp of the latest updated data of the service interface;

Wherein: the interface update screening module is used to send a polling request to the server, and receive an interface in which the data synchronization timestamp of the client interface returned by the server is inconsistent with the corresponding data update timestamp in the data synchronization control table in the server ;

The interface data request module calls the interface received by the above-mentioned interface update screening module and requests synchronization data from the server, and receives the data returned by the server to update the original data.

The data synchronization of the present invention needs to be completed by the server and the client. A data synchronization control table is established on the server, and then the server provides a polling interface for the client to send a request for connection. When the client polls, the local interface is first The data synchronization timestamp is the same as the corresponding data update timestamp in the data synchronization control table of the server. If the actual data synchronization timestamp is inconsistent with the data update timestamp, it means that there is a data update on the interface, and the server returns an inconsistent interface ID to the client. , and then the client requests data updates from the server by calling the inconsistent interface.

The technical scheme of the present invention is described below by a development example:

Taking the cloud POS project of one of the retail brands as an example, there are about 500,000 retailers in the country, and each retailer has only one cloud POS machine, each cloud POS machine has 20 data download interfaces, and each data download interface Corresponding types of data, such as commodity data, inventory data, user data, institutional parameters, category information, activity information, etc., and each cloud POS has an online time of 10 hours per day. If the traditional short polling method is used, each The interface must periodically request synchronization data from the server, and the server has to accept 600 million requests every day. Since the modification of each interface is small, these requests are all invalid requests. The above scheme is optimized with reference to the method of the present invention:

First, add a data table on the server side, the data columns in the data table are user code, interface ID and timestamp respectively, and the timestamp is the time point of the latest data update of the interface; when the merchant modifies the data in the background, The timestamp of the interface on the index table corresponding to the modified data will be updated;

When the business data changes, the records in the data synchronization control table should be updated at the same time. For example, if the commodity information is modified in the commodity management function, the timestamp of the user's corresponding commodity download interface should be modified in the program to be consistent with the timestamp in the commodity table. In order to reduce complexity To improve performance, reduce errors and prevent lock tables, use message queue (MQ) to asynchronously modify timestamps in data synchronization control tables.

Then add a polling interface on the server side for each cloud POS to make link requests at regular intervals, but the polling interface is not used for interface synchronization data, the parameters of this interface are each interface ID on the cloud POS machine and the interface The timestamp of the latest data synchronization of the ID. The function is the interface ID of the cloud POS machine. The timestamp of the latest synchronized data is compared with the timestamp of the corresponding interface data in the data table. The return value is the interface ID of the client whose comparison result is inconsistent. When there is a difference, the timestamp of the latest data synchronization of the interface on the cloud POS machine will be earlier than the timestamp recorded in the corresponding data row in the data table, indicating that there is data update under the interface ID, and then the cloud POS machine will call the returned interface. ID requests data update from the server. For example, there are 20 data download interfaces in the cloud POS machine. The cloud POS machine requests connection from the server through the polling interface. There are two interfaces with different timestamps, which means that only the data of these two interfaces has changed. The server returns the interface IDs of these two interfaces to the cloud POS machine. The cloud POS machine only needs to call these two interfaces to obtain the latest data from the server. The data is enough, eliminating the need for 18 invalid calls. Scaling to 500,000 retail customers can save a lot of invalid data requests every day.

The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims (8)

1. a data synchronization method, is characterized in that: described method comprises the steps: S1, establish a data synchronization control table on the server, the data synchronization control table records the data update timestamp of the latest data update performed by each interface, and when the server updates data, the data synchronization control table updates the corresponding data update timestamp; S2. The server provides a polling interface to the client. The parameters of this interface are the interface of the client and the data synchronization timestamp of the latest synchronization data of each interface. The function is the data synchronization timestamp of the client interface and the data synchronization control table. If there is inconsistency, the inconsistent interface is returned to the client, and the client calls the returned interface to request data synchronization from the server. 2. data synchronization method according to claim 1, is characterized in that: in described data synchronization control table, comprising data as the data update time stamp of user code, interface ID and latest data update, user code is used for corresponding customer When the server accepts manual data updates, the data update timestamp of the docking interface ID in the data synchronization control table is changed to the point in time when the data is updated. 3. data synchronization method according to claim 1 is characterized in that: in step S1, when server carries out data update, by establishing message queue, the data update time stamp of different interfaces and other data-related data tables are carried out. renew. 4. data synchronization method according to claim 2 is characterized in that: in step S2, the data synchronization time stamp of the local interface of the server client is compared with the data update time stamp in the data synchronization control table, and the data The interface ID whose synchronization timestamp is inconsistent with the data update timestamp is returned to the client. 5. The data synchronization method according to claim 4, characterized in that: in step S2, the process that the client fetches the returned interface and requests data synchronization from the server comprises: S201, the client invokes the interface ID returned by the server and requests synchronization data from the server; S202, the server returns the requested data to the corresponding interface, the corresponding interface data is updated, and the data synchronization timestamp of the interface is updated. 6. A data synchronization system, characterized in that: comprising a server and a client, the server comprising a data synchronization control table, a comparison module and a data transmission module; Wherein, the data synchronization control table is used to record the interface served by the server and the data update timestamp of the latest data update of each interface; The comparison module is used to compare the data synchronization timestamp of the latest synchronization data of the client interface with the corresponding data update timestamp in the data synchronization control table after accepting the corresponding request of the client, and output the two timestamps to the client that are inconsistent. Interface; The data sending module is used for the interface data synchronization request of the interface client, and sends corresponding update data to the interface. 7. The data synchronization system according to claim 6, characterized in that: the server further comprises a data update module, when the server performs data update, and the update of the same data involves the data update timestamp update of multiple interfaces When , the data update module updates the time stamps of different interfaces by establishing a message queue. 8. A data synchronization system, characterized in that it comprises a client and a server, the client includes an interface update screening module and an interface data request module, the server is provided with a data synchronization control table, and the data synchronization The control table is used to record the timestamp of the latest updated data of the service interface; Wherein: the interface update screening module is used to send a polling request to the server, and receive an interface in which the data synchronization timestamp of the client interface returned by the server is inconsistent with the corresponding data update timestamp in the data synchronization control table in the server ; The interface data request module invokes the interface received by the above-mentioned interface update screening module and requests synchronization data from the server, and receives the data returned by the server to update the original data.

Images