CN111813550A - Data processing method, device, server and storage medium - Google Patents

Abstract

The application discloses a data processing method, a device, a server and a storage medium, the scheme of the application is applied to a business service platform, the business service platform runs a master process and at least one slave process, and the method comprises the following steps: acquiring a data processing request sent by a client; in the case that the data processing request comprises a read request, determining a first data identification of data requested to be read by the read request; if the first data identifier belongs to a data identifier in an unsynchronized record of the cache, forwarding the reading request to the main process for processing, wherein the unsynchronized record records a data identifier corresponding to data which is not synchronized from the main process to the slave process; if the first data identification does not belong to the data identification in the unsynchronized record, the read request is forwarded to the slave process for processing. The scheme of the application can improve the data request processing efficiency and simultaneously reduce the data reading error or abnormal condition caused by the inconsistency of the master process data and the slave process data.

Description

Data processing method, device, server and storage medium

technical field

The present application relates to the technical field of data processing, and in particular, to a data processing method, device, server and storage medium.

Background technique

In business scenarios involving a large amount of data and high requirements for data read and write performance, the business service platform generally caches data through processes and processes data read and write requests to achieve more efficient data read and write processing.

At present, there are two common ways for the business service platform to process data read and write requests through processes: one is a single process method, and the other is a master-slave process method. The single-process mode refers to processing read requests and write requests at the same time through a single process. The advantage of the single-process mode is that there is no need for data synchronization, so there is no data inconsistency. However, due to the high read and write pressure of a single process, the client request delay is long. In the master-slave process mode, the business service platform will run a master process and at least one slave process. Among them, the master process is responsible for processing the client's write request and synchronizing the data to the slave process. On this basis, the slave process is responsible for processing the client's read request, which can reduce the data read and write pressure of the master process and help improve data read and write. efficiency and reduce request processing delay. However, in the master-slave process mode, the data of the master process and the slave process may be inconsistent due to the untimely synchronization of data from the master process to the slave process, so that the data fed back from the slave process to the client is incorrect or the data is read abnormally. It can be seen that how to reduce the data read error or exception caused by data inconsistency while reducing the request processing delay is a technical problem that those skilled in the art urgently need to solve.

SUMMARY OF THE INVENTION

In view of this, the present application provides a data processing method, device, server and storage medium, so as to improve the data request processing efficiency and reduce the data request processing delay, while reducing the data inconsistency between master and slave process data. Data read errors or exceptions.

To achieve the above purpose, the application provides the following technical solutions:

On the one hand, the present application provides a data processing method, which is applied to a business service platform. The business service platform runs a master process and at least one slave process, and the method includes:

Obtain the data processing request sent by the client;

In the case that the data processing request includes a read request, determining a first data identifier of the data requested by the read request;

If the first data identifier belongs to the data identifier in the cached unsynchronized record, the read request is forwarded to the master process for processing, and the unsynchronized record records that the master process has not yet been synchronized to the slave The data identifier corresponding to the data of the process;

If the first data identifier does not belong to the data identifier in the unsynchronized record, the read request is forwarded to the slave process for processing.

In a possible implementation, the method further includes:

In the case that the data processing request includes a write request, determining a second data identifier of the data requested to be written by the write request;

The write request is forwarded to the main process for processing, and the second data identifier is added to the unsynchronized record.

In yet another possible implementation, it also includes:

Obtain the synchronization completion indication fed back by the main process, the synchronization completion indication carries a third data identifier, and the synchronization completion indication is used to represent that the main process has synchronized the data represented by the third data identifier to the Describe the slave process;

Delete the third data identifier in the unsynchronized record.

In yet another possible implementation manner, the business service platform runs with multiple slave processes;

The method also includes:

In the case of confirming that the master process is faulty, query the synchronization sequence numbers stored by the multiple slave processes, wherein the synchronization sequence numbers represent the total amount of data synchronized from the master process to the slave processes;

determining, based on the synchronization sequence numbers stored by the multiple slave processes, the target slave process that obtains the largest total number of synchronization data from the master process;

The target slave process is determined to be the master process.

In another aspect, the present application also provides a data processing device, which is applied to a business service platform, where the business service platform runs a master process and at least one slave process, and the device includes:

The request obtaining unit is used to obtain the data processing request sent by the client;

a first identification determination unit, configured to determine a first data identification of the data requested by the read request when the data processing request includes a read request;

The first request processing unit is configured to forward the read request to the main process for processing if the first data identifier belongs to the data identifier in the unsynchronized record of the cache, and the unsynchronized record records the data that has not been retrieved from the cache. the data identifier corresponding to the data of the master process synchronized to the slave process;

A second request processing unit, configured to forward the read request to the slave process for processing if the first data identifier does not belong to the data identifier in the unsynchronized record.

In yet another aspect, the present application also provides a server, including a memory and a processor;

Wherein, the memory is used to store programs;

The processor is configured to execute the program, and when the program is executed, it is specifically configured to implement the data processing method according to any one of claims 1 to 6.

In another aspect, the present application also provides a storage medium for storing a program, which, when executed, is used to implement the data processing method described in any one of the above.

It can be seen from the above that the business service platform of the present application caches an unsynchronized record, and the unsynchronized record stores the data identifier corresponding to the data that has not been synchronized from the master process to the slave process. Based on this, after the application receives the read request from the client, only if the unsynchronized record does not include the data identifier requested by the read request, that is, it is confirmed that the data requested by the read request has been synchronized by the master process to the slave process. , the read request will be forwarded to the slave process for processing; and if the unsynchronized record includes the data identifier requested by the read request, the read request will still be forwarded to the master process for processing, thus minimizing the amount of time the master process handles. Under the premise of increasing the number of read requests, the data read errors or exceptions caused by data inconsistency between the master process and the slave process are reduced, thereby improving the request processing efficiency and reducing data read errors or exceptions. .

Description of drawings

In order to explain the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without any creative effort.

FIG. 1 shows a schematic diagram of the composition structure of a scenario to which the present application applies;

FIG. 2 shows a schematic flowchart of an embodiment of the data processing method of the present application;

FIG. 3 shows a schematic diagram of an implementation principle framework of the data processing method of the present application;

FIG. 4 shows a schematic diagram of flow interaction of another embodiment of the data processing method of the present application;

Fig. 5 shows a kind of schematic flow chart that the routing process selects the master process from multiple slave processes in the present application;

Fig. 6 shows an exemplary schematic diagram of the routing process selecting the main process;

FIG. 7 shows a schematic diagram of a composition structure of the data processing apparatus of the present application;

FIG. 8 shows a schematic diagram of a composition structure of the server of the present application.

Detailed ways

The data processing method of the present application is suitable for business service platforms in various business scenarios. For example, if the business scenario is a game business scenario, the business service platform is a game service platform, and accordingly, the game client can request the game service platform to read or write game-related data. For another example, in the case where the business scenario is financial transaction business, the business service platform is the transaction service platform. In this scenario, when the client performs payment or transfer services, it may involve sending money to the business service platform. Request to read or write data. Of course, the business scenario to which this application applies may also have other possibilities, which are not limited.

In this application, the business service platform may be an independent server, or may be a cloud platform, a server cluster, or a distributed system composed of multiple servers, or the like.

In this embodiment of the present application, a master process for processing read or write requests and at least one slave process for assisting the master process in processing read requests run on the business service platform.

In the embodiment of the present application, after receiving the data processing request from the client, the business service platform can reasonably allocate the data processing request to the master process in combination with the type of the data processing request and the data synchronization status between the master process and the slave process Or from the process, to reduce data read errors or read exceptions while ensuring the processing efficiency of data processing requests.

In a possible implementation manner, the business service platform also runs a routing process, and the routing process is a process independent of the master process and the slave process, which is responsible for managing the master process and the slave process, and assigns the master process and the slave process. read request or write request.

Wherein, when the business service platform includes multiple servers, the routing process, the master process and at least one slave process may run on the same server, or may run on different servers.

Optionally, in order to improve disaster tolerance (for example, to avoid a single server failure that affects multiple processes in the master process and slave process at the same time, etc.), or to ensure the running performance of each process, the routing process, the master process And each slave process can run on different servers respectively.

For ease of understanding, the following takes the business service platform as a cloud platform as an example to illustrate a scenario to which the solution of the present application is applicable, as shown in FIG. 1 .

As can be seen from FIG. 1 , the scenario includes: a cloud platform 10 , and the cloud platform may include multiple cloud servers 101 .

Among them, a cloud platform is also called a cloud computing platform, which is a network platform constructed based on cloud technology. Among them, cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data.

Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business models. Background services of technical network systems require a lot of computing and storage resources, such as image storage and encoding. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the back-end system for logical processing. Data of different levels will be processed separately, and all kinds of industry data need to be strong. The system backing support can only be achieved through cloud computing.

Among them, cloud computing is a computing mode that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides the resources is called the "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for according to usage.

As a basic capability provider of cloud computing, it will establish a cloud computing resource pool (referred to as cloud platform, generally called IaaS (Infrastructure as a Service) platform, and deploy various types of virtual resources in the resource pool for External customers choose to use. The cloud computing resource pool mainly includes: computing devices (which are virtualized machines, including operating systems), storage devices, and network devices.

As shown in FIG. 1 , a master process and at least one slave process run on the cloud platform 10 . In addition, routing processes other than the master process and the slave process run on the cloud platform. Wherein, the master process, the routing process and the at least one slave process may respectively run on different cloud servers of the cloud platform.

The scenario in FIG. 1 also includes at least one client 20, and the client can send a data processing request to the cloud platform, and the data processing request can be a write request or a read request.

Correspondingly, the cloud platform responds to the data processing request through the routing process, and determines the process for processing the data processing request from the master process and the slave process.

It can be understood that FIG. 1 takes the business service platform as a cloud platform as an example for illustration. In practical applications, the business service platform is a server cluster or other situations are similar, and details are not repeated here.

In combination with the above content, the image processing method of the present application will be introduced below in combination with the flowchart.

As shown in FIG. 2 , which shows a schematic flowchart of an embodiment of a data processing method, the method of this embodiment can be applied to a business service platform, for example, it can be specifically executed by a routing process of the business service platform. As can be seen from the foregoing, the business service platform runs with a master process and at least one slave process.

The method of this embodiment may include:

S201, a data processing request sent by a client is obtained.

The data processing request may include one or both of a write request and a read request. Among them, the write request is used to request to write data; the read request is used to request to read data. It is understandable that in most business scenarios, the data processing request is generally one of a write request and a read request.

S202, in the case that the data processing request includes a read request, determine a first data identifier of the data requested by the read request.

It can be understood that a read request and a write request will have different identification methods. Therefore, by analyzing the data processing request, it can be analyzed whether the data processing request includes a read request.

The data identifier of the data is used to uniquely identify the data. Among them, in different business scenarios, the specific form or composition structure of the data identifier that uniquely identifies the data will also be different, which can be set as required.

For example, taking the business scenario as a game scenario as an example, a piece of data can be uniquely identified by the data type and serial number of the data. For example, taking a gang game scene as an example, an organization composed of multiple players in a gang game can be called a gang. Therefore, the game data generated by any gang can be uniquely identified by the gang data type and the gang identity.

In this embodiment of the present application, for the convenience of distinction, the data identifier corresponding to the data requested by the read request is referred to as the first data identifier.

It is understandable that, in order for the business service platform to confirm the data requested by the read request, the read request will contain or carry the relevant information of the data identifier used to represent the data. Therefore, by parsing the read request, it can be parsed out. A first data identifier used to characterize the data to be read.

S203, if the first data identifier belongs to a data identifier in a cached unsynchronized record, forward the read request to the main process for processing.

Wherein, the unsynchronized record records the data identifier corresponding to the data that has not been synchronized from the master process to the slave process.

The specific forms of the unsynchronized records and the record data identifiers in the unsynchronized records may not be limited. For example, an unsynchronized record can directly store data identifiers; for another example, an unsynchronized record can include records corresponding to different data identifiers, and the records corresponding to each data identifier include not only the data identifier, but also the client request to write the data identifier. At the time of the data corresponding to the data identifier, other data related to the data identifier may also be recorded as required.

The business service platform may cache the unsynchronized process in a fixed cache space. For example, if the business service platform manages the master process and the slave process through the routing process, the unsynchronized record may be cached in the memory space corresponding to the routing process. .

Among them, there are many ways for the business service platform to determine the unsynchronized records. For ease of understanding, several possible situations are introduced.

In a possible situation, the business service platform may query the slave process for the data identifier of the data that has not been synchronized to the slave process according to the polling cycle or irregularly. Correspondingly, the business service platform can record the data identifier corresponding to the data that has not been synchronized from the master process to the slave process into the unsynchronized record; or, based on the queried data identifier, update the unsynchronized record.

However, in this case, the business service platform needs to continuously query the master process to synchronize data with the slave process, which may increase the occupation of bandwidth resources between the business service platform and the master process, and may also occupy the data processing of the master process. resource.

In order to reduce bandwidth occupation and avoid occupation of data processing resources of the main process, in another possible situation, when receiving a write request, the business service platform may determine the second data of the data requested to be written by the write request identifier, and the second data identifier is added to the unsynchronized record. Certainly, the business service platform also forwards the write request to the main process for processing while or after adding the second data identifier to the unsynchronized record.

It can be understood that when the business service platform receives the write request, since the data requested by the write request has not been written by the master process, the data corresponding to the second data identifier has not been synchronized from the master process to the slave process. process. It can be seen that after the business service platform receives the write request, it can determine the data requested by the write request to be the data that has not been synchronized from the master process to the slave process, and store the second data identifier in the write request to the unsynchronized data. Synchronize records, so that there is no need to query the main process for the data identifier corresponding to the data that has not been synchronized.

The above is an example of the two manners in which the business service platform generates unsynchronized records, and other manners that may determine the data identifier corresponding to the data that has not yet been synchronized are also applicable to this embodiment, which is not limited.

It can be understood that, in order to ensure that the data that the master process has not synchronized to the slave process is consistent with the data represented by the data identifier recorded in the unsynchronized records cached by the business service platform, the business service platform can also continuously update the unsynchronized records. , to delete the data ID in the unsynchronized record to complete the synchronization.

For example, in a possible implementation manner, the business service platform may periodically send a synchronized information request to the master process, and the synchronized information request is used to request the master process to feed back the corresponding data that has been synchronized to the slave process in the current cycle data identification. Among them, after the master process successfully synchronizes data to the slave process, it will store the data identifier corresponding to the synchronized data. Correspondingly, after the main process receives the synchronized information request, the main process sends the information of the data identifier corresponding to the synchronized data in the latest cycle to the business service platform. On this basis, the business service platform deletes the corresponding data identifiers in the unsynchronized records based on the data identifier information fed back by the main process.

This implementation method is actually that the business service platform periodically updates the unsynchronized records. Therefore, there may be cases where the unsynchronized records are not updated in a timely manner, resulting in a certain data has been synchronized from the master process to the slave process, but the business service platform The unsynchronized record still indicates that the data has not been synchronized to the slave process, so the read request for the data is still forwarded to the master process for processing, so that the number of read requests processed by the master process is relatively large.

In order to minimize the number of read requests handled by the main process, minimize the pressure on the main process, further improve the processing efficiency of data processing requests, and reduce the delay of client data processing requests, the application can also After the master process synchronizes the data to the slave process, it actively reports the data identifier corresponding to the synchronized data to the business service platform.

Specifically, the business service platform can obtain the synchronization completion indication fed back by the main process. The synchronization completion indication carries a third data identifier, and the synchronization completion indication is used to indicate that the master process has synchronized the data represented by the third data identifier to the slave process.

The synchronization completion indication is generated after the master process confirms that the data identified by the third data has been synchronized to the slave process. It can be understood that, for the convenience of distinction, the data identifier corresponding to the data that the master process has synchronized to the slave process is referred to as the third data identifier. Correspondingly, the business service platform may delete the third data identifier in the unsynchronized record based on the synchronization completion indication.

In the second method of updating unsynchronized records, each time the master process finishes synchronizing data to the slave process, it will notify the business service platform, so that the unsynchronized data determined based on the unsynchronized records and the master process that has not yet been sent to the slave process The synchronized data is consistent, which is beneficial to forward all read requests corresponding to the synchronized data to the slave process for processing, which minimizes the number of read requests that the master process needs to process and reduces the data processing of the master process. pressure.

It can be understood that if there is a data identifier of a certain data in the unsynchronized record, it means that the data has not been synchronized from the master process to the slave process, that is, the data is only cached in the memory space corresponding to the master process. Therefore, in the case where the first data identifier carried in the read request belongs to the data identifier in the unsynchronized record, in order to avoid data read and write errors or exceptions due to inconsistencies in the data represented by the first data identifier in the master process and the slave process , the read request will be assigned to the main process for processing.

S204, if the first data identifier does not belong to the data identifier in the unsynchronized record, forward the read request to the slave process for processing.

It can be understood that if the first data identifier of the data requested by the read request does not belong to the unsynchronized record, it can be determined that the data has been synchronized from the master process to the slave process. Therefore, in order to avoid increasing the pressure on the master process, you can Send the read request to the slave process for processing. Moreover, since the data in the master process and the slave process have been synchronized, there will be no data read error or exception due to inconsistency of the data in the master process and the slave process.

It can be understood that, when the business service platform runs multiple slave processes, the business service platform can select one slave process from the multiple slave processes, and forward the read request to the selected slave process. There are various ways for the service server platform to select the slave process for processing the read request. For example, a slave process with a smaller load can be selected to process the read request based on the load status of the slave process, which is not limited.

It can be seen that since the business service platform of the present application caches an unsynchronized record, the unsynchronized record stores the data identifier corresponding to the data that has not been synchronized from the master process to the slave process. Based on this, after the application receives the read request from the client, only if the unsynchronized record does not include the data identifier requested by the read request, that is, it is confirmed that the data requested by the read request has been synchronized by the master process to the slave process. , the read request will be forwarded to the slave process for processing; and if the unsynchronized record includes the data identifier requested by the read request, the read request will still be forwarded to the master process for processing, thus minimizing the amount of time the master process handles. Under the premise of increasing the number of read requests, the data read errors or exceptions caused by data inconsistency between the master process and the slave process are reduced, thereby improving the request processing efficiency and reducing data read errors or exceptions. .

For ease of understanding, the following is an example of a business service platform running a routing process, and scheduling data processing requests to the master process and the slave process through the routing process.

As shown in FIG. 3 , it shows a schematic frame diagram of an implementation principle of the data processing method of the present application.

In FIG. 3 , the business service platform 300 runs a routing process 301 , a master process 302 and at least one slave process 303 .

As shown in Figure 3, there are unsynchronized records cached in the routing process.

Wherein, after the client 310 sends the read request or the write request to the business service platform, the routing process of the business service platform will obtain the read request or the write request sent by the client.

Wherein, for the obtained write request, the routing process 301 may directly send the write request to the main process.

For the obtained read request, the routing process will detect whether the cached unsynchronized record is hit, that is, whether the data identifier carried in the read request belongs to the data identifier in the unsynchronized record. Correspondingly, the routing process will send the read request that misses the unsynchronized record to the slave process; if the read request hits the unsynchronized record, it will send the read request to the master process for processing.

The data processing method of the present application will be described in detail below on the basis of the principle frame diagram of FIG.

The data processing method shown in Figure 4 may include:

S401, the client sends a read request to the business service platform, so that the routing process obtains the read request.

It can be understood that, in this embodiment, the data processing request sent by the client to the business service platform may be either a read request or a write request. If the data processing request sent by the client to the business service platform includes a read request and a write request, the routing process can separate these two requests and perform corresponding processing on the read request and the write request respectively.

S402, the routing process obtains the first data identifier carried in the read request.

Wherein, the first data identifier is used to represent the data requested to be read by the read request.

S403, if the routing process confirms that the first data identifier belongs to the data identifier in the cached unsynchronized record, the routing process forwards the read request to the main process for processing.

It can be understood that after receiving the read request, the main process can query corresponding data from the cached service data based on the read request, and then return the queried data to the client. There is no restriction on the specific implementation of the main process returning corresponding data to the client based on the read request, and this part of the content is not shown in FIG. 4 .

S404, if the routing process confirms that the first data identifier does not belong to the data identifier in the cached unsynchronized record, the routing process selects a slave process for processing the read request from at least one slave process, and forwards the read request to the selected slave process out of the slave process.

Similar to the master process processing read requests, since the slave process also caches various data synchronized by the master process, the slave process can respond to the read request, query the data corresponding to the read request from the cached data, and send the queried data. Data is returned to the client.

S405, the client sends a write request to the business service platform, so that the routing process obtains the write request sent by the client.

S406, the routing process confirms the second data identifier of the data requested to be written by the write request, and adds the second data identifier to the unsynchronized record in the cache.

It is understandable that before the write request is forwarded to the master process, the master process has not written the data corresponding to the write request. Therefore, the data naturally does not belong to the data that has been synchronized to the slave process. Therefore, the routing process can determine that the data is Data that has not been synchronized by the master process to the slave process, so that the second data identifier of the data is stored in the unsynchronized record.

As an optional manner, adding the second data identifier to the unsynchronized record may be to generate a record corresponding to the second data identifier. For example, the second data identifier is stored in the record corresponding to the second data identifier. The record corresponding to the second data identifier may also store other data related to the second data identifier.

It can be understood that, in any of the above embodiments of the present application, considering that the space for caching unsynchronized records in the business service platform is relatively limited, for example, the memory space allocated by the business service platform to the routing process is limited, therefore, if the business If the number of data processing requests processed by the service platform is large, the number of data identifiers that may need to be stored in the unsynchronized record will also be relatively large.

In this case, when the business service platform confirms that the cache space is insufficient, the application can also delete the long-unused data identifiers in the unsynchronized records. Wherein, the long-term unused data identifier in the unsynchronized record refers to the data identifier of the data that has not been requested to be read for a relatively long time.

Specifically, after the routing process obtains a read request for the second data identifier (that is, a read request requesting to read the data corresponding to the second data identifier), if the second data identifier belongs to the unsynchronized record, the routing process also The time when the read request is obtained is determined as the access time of the second data identifier, and the access time is stored in the record corresponding to the second data identifier.

Wherein, in the case where the access time is stored in the record of the second data identifier, the routing process will use the last determined access time to update the access time in the record of the second data identifier, so that the access time of the second data identifier is updated. Stored in the record is the latest access time of the second data identifier.

Correspondingly, if the routing process confirms that the cache space is insufficient, it can delete at least one data identifier record whose access time is longer from the current time in the unsynchronized records. For example, delete the records of each data identifier whose access time is longer than the current time, or delete the records of one or more data identifiers whose access time is the longest from the current time.

S407, the routing process forwards the write request to the main process for processing.

It should be noted that the sequence of the two steps of adding the second data identifier to the cached unsynchronized record and forwarding the write request to the main process is not limited, for example, these two steps may also be performed simultaneously.

S408, the main process performs a data write operation based on the write request, and caches the write operation result.

The main process executes the write operation in response to the write request, thereby implementing the cache of the data corresponding to the write operation.

S409, the master process sends at least one piece of data that has been written but not yet synchronized to each slave process.

The specific manner in which the master process sends the data that needs to be synchronized to the slave process may not be limited. For example, the master process may send the written data to each slave process after performing the write operation corresponding to the write request, so as to synchronize the data to the slave process. For another example, in order to avoid frequently synchronizing data and occupying too many processing resources of the main process, the main process can also periodically (such as according to a set time interval or period, etc.) send a data synchronization instruction to the main process, and the data synchronization instruction can carry the main process. All data that the process has written to the cache space of the master process but not synchronized to the slave process. Of course, the master process may also have other ways of synchronizing data with the slave process, which will not be repeated here.

S410: After confirming that the at least one piece of data is successfully synchronized to the slave process, the master process confirms the respective third data identifiers of the at least one piece of data successfully synchronized, and feeds back a synchronization completion indication corresponding to the at least one piece of data to the routing process.

Among them, there may also be various ways for the master process to confirm whether the data is successfully synchronized with the slave process. For example, after the slave process completes data synchronization based on the data sent by the master process, it can feed back a success indication to the master process, and the success indication indicates that the slave process has completed synchronization of corresponding data. For example, the success indication carries the data identifier of the data whose synchronization is successful. Of course, the master process can actively query the synchronization status of the slave process to detect whether the data synchronization is successful.

S411, the routing process deletes the at least one third data identifier in the unsynchronized record based on the at least one third data identifier carried in the synchronization completion indication.

After obtaining the synchronization completion instruction fed back by the main process, the routing process can determine that the synchronization completion instruction carries the data represented by the third data identifier and has completed data synchronization. On this basis, the third data identifier can be deleted from the unsynchronized record. . Correspondingly, after subsequently receiving the read request carrying the third data identifier, the read request can be forwarded to the slave process for processing.

It is understandable that if the main process in the business service platform fails, it may affect the business service platform's processing of client read and write requests. Therefore, after it is determined that the main process is faulty, corresponding recovery is required. mechanism. However, if the main process is restarted directly, it will be complicated and time consuming. In order to solve this problem, after it is determined that the master process is faulty, the business service platform of the present application will reselect a slave process as the master process from multiple slave processes.

As shown in FIG. 5, it shows a schematic flowchart of the routing process of the present application selecting a master process from multiple slave processes. This embodiment may include:

S501 , in the case that the heartbeat packet sent by the master process is not detected for a set period of time, the routing process queries the synchronization sequence numbers stored in each slave process.

The main process may send a heartbeat packet to the routing process according to the set first heartbeat cycle. Correspondingly, the slave process may send a heartbeat packet to the routing process according to the set second heartbeat period. The first heartbeat period and the second heartbeat period may be the same or different.

For example, the master process or the slave process can send a heartbeat packet to the business service platform according to the corresponding heartbeat cycle when the heartbeat sending time is reached.

It is understandable that since both the master process and the slave process periodically send heartbeat packets to the routing process, the routing process can analyze whether there is an abnormality in the master process based on the heartbeat packets sent by the master process; correspondingly, the routing process can also Based on the heartbeat packets sent by the slave process, analyze whether there is an abnormality in the slave process.

If the routing process does not receive the heartbeat packet of the main process for a set period of time, the routing process confirms that the main process is faulty.

It can be understood that, in this embodiment, whether the main process is faulty is analyzed based on the fact that the main process sends a heartbeat packet to the routing process. However, it is understandable that in practical applications, the routing process can also find the fault of the main process in other ways. For example, the routing process can periodically send detection packets to the main process. If the feedback of the detection packet, or the feedback of multiple consecutive detection reports is received, it is confirmed that the main process is faulty.

The synchronization sequence number represents the total amount of data synchronized from the master process to the slave process. The total number of synchronization data from the master process to the slave processes may also be considered as the total number of rounds of synchronization data initiated by the master process to multiple slave processes. Wherein, each round of data synchronization corresponds to a round of data synchronization instructions initiated by the master process to multiple slave processes for one data to be synchronized.

In a possible implementation manner, each time the master process synchronizes data with the slave process, the synchronization sequence number will be updated, so that the total amount of synchronization data from the master process to the slave process represented by the synchronization sequence number increases. For example, the synchronization sequence number can be the amount of data synchronized from the master process to the slave process, and each time the synchronization sequence number is updated, the amount is incremented by one. Correspondingly, the master process can send the data to be synchronized together with the latest updated synchronization sequence number to each slave process each time, so that the slave process stores the synchronization sequence number.

In another possible implementation manner, each time the master process receives a write request, it updates the synchronization sequence number, so that the total amount of synchronization data from the master process to the slave process represented by the synchronization sequence number increases. Correspondingly, when the master process synchronizes data with the slave process, the data to be synchronized may be sent to each slave process together with the currently determined synchronization sequence number.

For example, after the main process receives the write request for the first time, the synchronization sequence number can be set to "0001", then after the main process completes the write operation of the write request, the main process can change the data corresponding to the write operation and the synchronization sequence number "0001" to one and sent to each slave process. After receiving the write request for the second time, the synchronization sequence number will be changed to "0002". In this way, after the master process completes the write operation of the write request received for the second time, it will synchronize the data to each slave process. At the same time, the synchronization sequence number "0002" is also sent to each process.

It is understandable that since the synchronization sequence number can reflect the total amount of data synchronized from the master process to the slave process, if the master process fails and some data in the master process is only synchronized to part of the slave processes, then according to the synchronization stored in the slave process The sequence number identifies which slave processes have synchronized data that are more complete.

For example, when the above master process sends synchronization data to each synchronization process for the second time, it carries the synchronization sequence number "0002", then if the slave process has received the second synchronization data from the master process, it will be stored in the slave process. The synchronization sequence number should be "0002". However, if the synchronization sequence number stored by the slave process is "0001", it means that the slave process has not completed the second data synchronization, but obtained the first synchronization data of the master process.

S502: Determine, based on the synchronization sequence numbers stored by the multiple slave processes, the target slave process that obtains the largest total number of synchronization data from the master process.

The target slave process refers to the slave process with the largest total number represented by the synchronization sequence numbers stored in the multiple slave processes. It can be understood that, in the case where there are at least two slave processes with the largest total number of synchronous data represented, one slave process can be randomly selected as the target slave process; it can also be based on the load situation, from the at least two slave processes. Select a target from the process.

For example, in the above example, the slave process with the stored synchronization sequence number "0002" obtains more total synchronization data from the master process than the slave process with the stored synchronization sequence number "0001".

S503, the target slave process is determined as the master process.

It can be seen from the above introduction that the data in the target slave process has the highest consistency with the data in the master process. Therefore, determining the new master process for the target slave process is beneficial to reduce data read exceptions caused by data inconsistency.

The operations required to determine the target slave process as the master process may include: storing the information of the target slave process as the master process; and may also include: broadcasting the target slave process as the master process to the target slave process and other slave processes Information.

It can be understood that after the target slave process becomes the new master process, the target slave process can synchronize the data stored by itself with other slave processes to ensure data consistency.

For ease of understanding, the total number of data synchronized by the master process to the slave process is taken as an example. For example, see FIG. 6 , which shows a schematic diagram of the principle that the routing process selects the master process from multiple slave processes.

When the master process is down in Figure 6, the synchronization sequence number stored in the master process is 112, which means that the master process has synchronized 112 pieces of data to the slave process (or initiated 112 rounds of data synchronization), but since the 112th round of data synchronization is synchronized During the process of the 112th data, the master process is down, so that only part of the slave processes complete the synchronization of the 112th data.

As shown in Figure 6, the synchronization sequence number in process 1 is 112, which means that process 1 has obtained the 112th data synchronized by the master process to the slave process 1, while the synchronization needs of process 2 and process 3 are still 111. Therefore, the process 20 Since the synchronization sequence number stored in the slave process 1 is the largest, it means that the data consistency between the slave process 1 and the faulty master process is the highest. Therefore, the slave process can be determined as the new master process.

Correspondingly, the new master process can synchronize its own data to other slave processes to ensure data consistency between each master process and the new master process.

Corresponding to a data processing method of the present application, the present application also provides a data processing apparatus.

As shown in FIG. 7 , it shows a schematic diagram of the composition and structure of an embodiment of a data processing apparatus of the present application. The apparatus of this embodiment is applied to a business service platform, and the business service platform runs a master process and at least one slave process, The device includes:

A request obtaining unit 701, configured to obtain a data processing request sent by the client;

A first identification determination unit 702, configured to determine the first data identification of the data requested to be read by the read request when the data processing request includes a read request;

The first request processing unit 703 is used for forwarding the read request to the main process for processing if the first data identifier belongs to the data identifier in the unsynchronized record of the cache, and the unsynchronized record records the data that has not been synchronized from the main process to the data identifier. The data identifier corresponding to the data of the slave process;

The second request processing unit 704 is configured to forward the read request to the slave process for processing if the first data identifier does not belong to the data identifier in the unsynchronized record.

In a possible implementation, the apparatus may further include:

a second identification determining unit, configured to determine a second data identification of the data requested to be written by the write request when the data processing request includes a write request;

A third request processing unit, configured to forward the write request to the main process for processing, and add the second data identifier to the unsynchronized record.

Optionally, the device further includes:

The instruction obtaining unit is used to obtain the synchronization completion instruction fed back by the main process, the synchronization completion instruction carries a third data identifier, and the synchronization completion instruction is used to represent that the main process has synchronized the data represented by the third data identifier to the slave process;

An identification deletion unit, configured to delete the third data identification in the unsynchronized record.

Optionally, when the second data identifier is added to the unsynchronized record, the third request processing unit is specifically configured to add the record of the second data identifier to the unsynchronized record;

Correspondingly, the device may also include:

The time recording unit is used to determine the time of obtaining the read request as the access time of the second data flag after obtaining the read request for the second data flag, and store the access time of the second data flag in the in the record of the second data identification;

The record deletion unit is used for, in the case of insufficient cache space, based on the addition time of each data mark in this unsynchronized record, to delete the record of at least one data mark whose duration is longer from this access moment from the current moment in this unsynchronized record .

In a possible implementation manner, the business service platform runs with multiple slave processes;

The device of the present application may also include:

a sequence number query unit, configured to query the synchronization sequence numbers stored by the multiple slave processes in the case of confirming that the master process is faulty, wherein the synchronization sequence number represents the total amount of data synchronized from the master process to the slave processes;

a process determination unit, configured to determine, based on the synchronization sequence numbers stored by the multiple slave processes, the target slave process that obtains the largest total number of synchronization data from the master process;

The master process replacement unit is used to determine the target slave process as the master process.

Optionally, when the sequence number query unit confirms that the main process is faulty, it is specifically: in the case that the heartbeat packet sent by the main process is not detected for a set period of time, confirming that the main process is faulty, wherein the main process is faulty. According to the set heartbeat period, when the heartbeat sending time arrives, a heartbeat packet is sent to the business service platform.

In another aspect, the present application also provides a server, where the server may be an independent server, a server in a server cluster, or a node in a cloud platform, or the like. FIG. 8 shows a schematic diagram of a composition structure of the server provided by the present application. In FIG. 8 , the server 800 may include: a processor 801 and a memory 802 .

Optionally, the server may further include: a communication interface 803 , an input unit 804 , a display 805 and a communication bus 806 .

The processor 801 , the memory 802 , the communication interface 803 , the input unit 804 and the display 805 all communicate with each other through the communication bus 806 .

In this embodiment of the present application, the processor 801 may be a central processing unit, an application-specific integrated circuit, or the like.

The processor may call the program stored in the memory 802, and specifically, the processor may execute the operations performed on the cloud server side in the above embodiments.

The memory 802 is used to store one or more programs, and the programs may include program codes, and the program codes include computer operation instructions. In this embodiment of the present application, the memory at least stores data for implementing any one of the above embodiments. Procedure for processing methods.

In a possible implementation manner, the memory 802 may include a storage program area and a storage data area, wherein the storage program area may store the operating system, the above-mentioned programs, and application programs required for functions such as image playback, etc. ; The storage data area can store data created according to the use of the server.

The communication interface 803 may be an interface of a communication module.

The present application may also include an input unit 804, which may include a touch sensing unit, a keyboard, and the like.

The display 805 includes a display panel, such as a touch display panel or the like.

Of course, the server structure shown in FIG. 8 does not constitute a limitation on the server in this embodiment of the present application. In practical applications, the server may include more or less components than those shown in FIG. 8 , or combine some components.

On the other hand, the present application also provides a storage medium, where computer-executable instructions are stored in the storage medium, and when the computer-executable instructions are loaded and executed by a processor, the data processing method in any of the above embodiments is implemented. .

According to one aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in any one of the above data processing method embodiments.

It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same and similar parts among the various embodiments, refer to each other Can. As for the apparatus type embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant part, please refer to the partial description of the method embodiment.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or device that includes the element.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, some improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be regarded as It is the protection scope of the present invention.

Claims (10)

1. a data processing method, is characterized in that, is applied to business service platform, described business service platform runs with master process and at least one slave process, described method comprises: Obtain the data processing request sent by the client; In the case that the data processing request includes a read request, determining a first data identifier of the data requested by the read request; If the first data identifier belongs to the data identifier in the cached unsynchronized record, the read request is forwarded to the master process for processing, and the unsynchronized record records that the master process has not yet been synchronized to the slave The data identifier corresponding to the data of the process; If the first data identifier does not belong to the data identifier in the unsynchronized record, the read request is forwarded to the slave process for processing. 2. The method of claim 1, further comprising: In the case that the data processing request includes a write request, determining a second data identifier of the data requested to be written by the write request; The write request is forwarded to the main process for processing, and the second data identifier is added to the unsynchronized record. 3. The method of claim 2, further comprising: Obtain the synchronization completion indication fed back by the main process, the synchronization completion indication carries a third data identifier, and the synchronization completion indication is used to represent that the main process has synchronized the data represented by the third data identifier to the Describe the slave process; Delete the third data identifier in the unsynchronized record. 4. The method according to claim 2, wherein the adding the second data identifier to the unsynchronized record comprises: adding a record of the second data identification to the unsynchronized record; The method also includes: After obtaining the read request for the second data identifier, the time when the read request is obtained is determined as the access time of the second data identifier, and the access time of the second data identifier is stored in the first data identifier. 2. In the record of data identification; In the case of insufficient cache space, based on the addition time of each data identifier in the unsynchronized record, delete the record of at least one data identifier whose access time is longer from the current time in the unsynchronized record. 5. The method according to claim 1, wherein the business service platform runs a plurality of slave processes; The method also includes: In the case of confirming that the master process is faulty, query the synchronization sequence numbers stored by the multiple slave processes, wherein the synchronization sequence numbers represent the total amount of data synchronized from the master process to the slave processes; determining, based on the synchronization sequence numbers stored by the multiple slave processes, the target slave process that obtains the largest total number of synchronization data from the master process; The target slave process is determined to be the master process. 6. The method according to claim 5, wherein the confirming that the main process is faulty comprises: In the case where the heartbeat packet sent by the main process is not detected for more than a set period of time, confirm that the main process is faulty, wherein the main process according to the set heartbeat cycle, when the heartbeat sending time arrives, to the business service The platform sends heartbeat packets. 7. A data processing device, characterized in that it is applied to a business service platform, wherein the business service platform runs a master process and at least one slave process, the device comprising: The request obtaining unit is used to obtain the data processing request sent by the client; a first identifier determination unit, configured to determine a first data identifier of the data requested by the read request when the data processing request includes a read request; The first request processing unit is configured to forward the read request to the main process for processing if the first data identifier belongs to the data identifier in the unsynchronized record of the cache, and the unsynchronized record records that the unsynchronized record has not yet been retrieved from the cache. the data identifier corresponding to the data of the master process synchronized to the slave process; A second request processing unit, configured to forward the read request to the slave process for processing if the first data identifier does not belong to the data identifier in the unsynchronized record. 8. The apparatus of claim 7, further comprising: a second identifier determining unit, configured to determine a second data identifier of the data requested to be written by the write request when the data processing request includes a write request; A third request processing unit, configured to forward the write request to the main process for processing, and add the second data identifier to the unsynchronized record. 9. A server, comprising a memory and a processor; Wherein, the memory is used to store programs; The processor is configured to execute the program, and when the program is executed, it is specifically configured to implement the data processing method according to any one of claims 1 to 6. 10 . A storage medium, characterized in that it is used for storing a program, and when the program is executed, it is used to implement the data processing method according to any one of claims 1 to 6 .

Images