CN104506643A - Server of distributed storage system and data synchronizing method between servers - Google Patents

Abstract

An embodiment of the present invention provides a server of a distributed storage system and a method for synchronizing data between servers. The method includes: the first server receives a data update request, and searches for the data component corresponding to the keyword information in the data update request. According to the update data in the data update request, the data fragment is updated; if it is detected that the second server with other data fragments with the same name as the data fragment is offline, the update operation of the updated data is recorded in the first server's In the first log, the first server and the second server both have data fragments with the same name; in response to the log synchronization request sent by the second server, return the first log to the second server. In the technical solution of the present invention, the second server only needs to perform data synchronization according to the first log, which greatly improves the efficiency of data synchronization, the server can provide data services to users as soon as possible, and the user waiting time caused by the server being offline can be reduced.

Description

Server of distributed storage system and data synchronization method between servers

technical field

The present invention relates to the technical field of distributed storage servers, in particular, the present invention relates to a server of a distributed storage system and a method for synchronizing data between servers.

Background technique

With the advent of the era of big data, service providers based on the Internet or communication networks need to store massive amounts of data to support their operations. At the same time, service providers can provide users with more convenient and Personalized service, so as to achieve the purpose of improving its service level.

At present, massive amounts of data are usually stored by distributed storage systems. For example, technicians distribute server clusters and store massive data in different servers according to certain rules; these servers can usually provide users with services such as data query and data update according to user requests. In order to improve data security and reliability, a distributed storage system is usually provided with a certain number of backup servers for backing up data.

However, in the existing data synchronization methods of distributed storage systems, the backup server usually does not provide services to users, but only copies the data in the hard disk of the current server to the backup server according to the specified time interval; After a fault restarts, the data in the entire backup server is usually transferred and restored to the current server, and more data is transferred, resulting in a longer time for data restoration. That is to say, the current server cannot respond to any data operation request related to the current server during the time interval from the failure to the device recovery, which will cause the user to wait for a long time after requesting the service and reduce the user experience. In addition, the existing data synchronization method is also likely to cause congestion of the local area network where the server and the backup server are located, affecting the normal operation of other servers or the backup server.

Therefore, the existing data synchronization methods of distributed storage systems have the defects that data synchronization takes a long time and consumes high network resources. Further, it may not be able to respond to user requests in time, causing users to wait for a long time , reducing the user experience.

Contents of the invention

Aiming at the shortcomings of the existing data synchronization method of the distributed storage system, the present invention proposes a data synchronization method between the servers of the distributed storage system and the servers to solve the problem of low efficiency existing in the existing digital synchronization method question.

According to one aspect, the technical solution of the present invention provides a data synchronization method for a first server in a distributed storage system, including:

The first server receives the data update request, and searches for data fragments corresponding to the keyword information in the data update request;

update the data fragments according to the update data in the data update request;

If it is detected that the second server configured with other data fragments with the same name as the data fragment is offline, the update operation of the updated data is recorded in the first log of the first server, wherein the second The first server and the second server are sibling servers, both of which have data fragments with the same name;

In response to the log synchronization request sent by the second server, return the first log to the second server.

Further, after returning the first log to the second server, it also includes:

receiving the data synchronization request sent by the second server, and parsing the keyword information;

Find the local data fragment corresponding to the parsed keyword information;

Return the found data in the data fragments to the second server.

Preferably, the first server receives a data update request, which specifically includes:

The master control process pre-created by the first server receives data update requests sent by multiple clients;

Based on each data update request received, generate a working channel for the data update request;

The main control process evenly distributes the received data update requests and corresponding working channels to multiple working processes pre-created by the first server;

The working process searches for corresponding data fragments according to the keyword information in the assigned data update request.

Preferably, the main control process receives data update requests sent by multiple clients, specifically including:

The main control process utilizes the HTTP protocol to monitor the network port;

The HTTP protocol is used to convert the format of the monitored data update request, and the converted data update request is used as the received data update request.

According to another aspect, the technical solution of the present invention also provides a second server-side data synchronization method in a distributed storage system, including:

Send a log synchronization request to the first server;

According to the received first log returned by the first server, a data synchronization operation is performed, wherein the first server and the second server are sibling servers, and data fragments with the same name are both set therein.

Preferably, a log synchronization request is sent to the first server, including:

When the second server is online, periodically sending a log synchronization request to the first server; or

When the second server changes from an offline state to an online state, a log synchronization request is sent to the first server.

Preferably, the data synchronization operation is performed according to the received first log returned by the first server, which specifically includes:

Comparing the received first log returned by the first server with the local second log to determine an update operation list;

According to the update operation list, update the corresponding local data fragments.

Preferably, the update operation list specifically includes a plurality of update operations for updating data; and

According to the update operation list, update the corresponding local data fragments, specifically including:

According to the chronological sequence of multiple update operations in the update operation list, the corresponding local data fragments are updated sequentially.

Preferably, according to an update operation, update the corresponding local data fragments, specifically including:

Based on the first log, determine keyword information recorded corresponding to the update operation;

carrying the determined keyword information in the data synchronization request and sending it to the first server;

Finding local data fragments corresponding to the keyword information;

Update the found local data fragments according to the received data returned by the first server.

According to another aspect of the technical solution of the present invention, a first server in a distributed storage system is also provided, including:

A data fragment determination module, configured to receive a data update request, and search for data fragments corresponding to the keyword information in the data update request;

A data update module, configured to update the data fragments according to the update data in the data update request;

An update operation recording module, configured to record the update operation of the update data in the first log of the first server if it is detected that the second server configured with other data fragments with the same name as the data fragment is offline In, wherein, the first server and the second server are sibling servers, both of which are provided with data fragments with the same name;

A synchronization request response module, configured to return the first log to the second server in response to the log synchronization request sent by the second server.

Further, the synchronization request response module is also used for receiving the data synchronization request sent by the second server, and analyzing the keyword information therefrom; searching for local data fragments corresponding to the parsed keyword information; The data is returned to the second server.

Preferably, the data fragmentation determining module specifically includes:

The data update request receiving unit is used to receive the data update requests sent by multiple clients by the pre-created main control process; based on each data update request received, generate the working channel of the data update request; the main control process Evenly distribute the received data update request and the corresponding working channel to multiple working processes pre-created by the first server;

The data fragment determination unit is configured to use the working process to search for corresponding data fragments according to the assigned keyword information in the data update request.

Preferably, the data update request receiving unit is specifically configured to monitor the network port by the main control process using the HTTP protocol; use the HTTP protocol to convert the format of the monitored data update request, and receive the converted data update request as received data update requests.

According to another aspect of the technical solution of the present invention, a second server in a distributed storage system is also provided, including:

A synchronization request sending module, configured to send a log synchronization request to the first server;

A data synchronization module, configured to perform a data synchronization operation according to the received first log returned by the first server; wherein, the first server and the second server are sibling servers to each other, and both of them are provided with data with the same name Fragmentation.

Preferably, the synchronization request sending module is specifically configured to periodically send a log synchronization request to the first server when the second server is in an online state; or send a log synchronization request to the first server when the second server changes from an offline state to an online state. A server sends a log synchronization request.

Preferably, the data synchronization module specifically includes:

An update operation list determining unit, configured to compare the received first log returned by the first server with the local second log to determine an update operation list;

A data update unit, configured to update the corresponding local data fragments according to the update operation list.

Preferably, the update operation list specifically includes a plurality of update operations for updating data; and

The data update unit is specifically configured to sequentially update corresponding local data fragments according to the time sequence of multiple update operations in the update operation list.

Preferably, the data update unit is further configured to determine the keyword information recorded corresponding to the update operation based on the first log; carry the determined keyword information in the data synchronization request and send it to the first The server; searching for local data fragments corresponding to the keyword information; updating the found local data fragments according to the received data returned by the first server.

In the technical solution of the present invention, the first server of the distributed storage system records the data update operation during the offline period of the second server into a log, and after the second server goes online, it compares the local data with the data of the first server according to the log. Synchronization; compared with the data synchronization method of updating all data in the second server with all backup data in the prior art, the data synchronization method of this program greatly reduces the amount of data that the second server needs to synchronize, and shortens the time required for data synchronization. time, improve the efficiency of data synchronization, and further enable the second server to provide data services to users in the shortest possible time after a fault event occurs, which can reduce user waiting time caused by server offline and improve user experience.

Moreover, in the technical solution of the embodiment of the present invention, a main control process and a work process are created in the server; the main control process is specially responsible for receiving the user's request, and evenly distributes the received request to a plurality of work processes, and the work process processes the request Response; the method described in this solution can achieve the purpose of greatly improving the load balance of the server and improving the response speed of the server to user requests due to more fully and efficiently utilizing server resources, thereby improving user experience.

Further, in the technical solution of the embodiment of the present invention, there is no need to add source codes for protocol or format conversion in the process of the server and the client, and the process can directly use the HTTP protocol to interact with the client, which improves the interaction between receiving user requests and the like. operational efficiency.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

FIG. 1a is a schematic flow diagram of a method for synchronizing data between servers in a distributed storage system according to an embodiment of the present invention;

FIG. 1b is a schematic diagram of an example of a data synchronization method according to an embodiment of the present invention;

Fig. 2a is a schematic flowchart of a method for updating local data fragments by a second server according to an update operation list according to an embodiment of the present invention;

Fig. 2b is a schematic diagram of the principle of a data acquisition method according to an embodiment of the present invention;

FIG. 3 is a schematic framework diagram of a server internal structure of a distributed storage system according to an embodiment of the present invention;

FIG. 4 is a schematic framework diagram of the internal structure of the data fragmentation determination module according to an embodiment of the present invention;

FIG. 5 is a schematic framework diagram of an internal structure of a data synchronization module according to an embodiment of the present invention.

Detailed ways

Those skilled in the art will understand that unless otherwise stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the word "comprising" used in the description of the present invention refers to the presence of said features, integers, steps, operations, elements and/or components, but does not exclude the presence or addition of one or more other features, Integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Additionally, "connected" or "coupled" as used herein may include wireless connection or wireless coupling. The expression "and/or" used herein includes all or any elements and all combinations of one or more associated listed items.

Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention belongs. It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have meanings consistent with their meaning in the context of the prior art, and unless specifically defined as herein, are not intended to be idealized or overly Formal meaning to explain.

Those skilled in the art can understand that the "terminal" and "terminal equipment" used here not only include wireless signal receiver equipment, which only has wireless signal receiver equipment without transmission capabilities, but also include receiving and transmitting hardware. A device having receiving and transmitting hardware capable of performing bi-directional communication over a bi-directional communication link. Such equipment may include: cellular or other communication equipment, which has a single-line display or a multi-line display or a cellular or other communication equipment without a multi-line display; PCS (Personal Communications Service, personal communication system), which can combine voice, data Processing, facsimile and/or data communication capabilities; PDA (Personal Digital Assistant, Personal Digital Assistant), which may include radio frequency receiver, pager, Internet/Intranet access, web browser, notepad, calendar and/or GPS (Global Positioning System , Global Positioning System) receiver; a conventional laptop and/or palmtop computer or other device having and/or including a radio frequency receiver. As used herein, a "terminal", "terminal device" may be portable, transportable, installed in a vehicle (air, sea, and/or land), or adapted and/or configured to operate locally, and/or In distributed form, the operation operates at any other location on Earth and/or in space. The "terminal" and "terminal device" used here can also be a communication terminal, an Internet terminal, a music/video player terminal, such as a PDA, a MID (Mobile Internet Device, a mobile Internet device) and/or a music/video player Functional mobile phones, smart TVs, set-top boxes and other devices.

Those skilled in the art should understand that the concepts of "application", "application program", "application software" and similar expressions referred to in the present invention are the same concepts well known to those skilled in the art, and refer to a series of computer instructions and related Computer software that is organically constructed from data resources and suitable for electronic operation. Unless otherwise specified, this naming itself is not limited by the type of programming language, level, or the operating system or platform on which it runs. Naturally, such concepts are also not limited by any form of terminal. Similarly, there is a corresponding relationship between the "target application" and "installation package" mentioned in the present invention, and the installation package is the file existence form of the target application.

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

The inventors of the present invention consider that the same data can be stored in data fragments with the same name in the first and second servers of the distributed storage system respectively. After the first server updates the data in the data fragment, if it detects that the second server is offline, it can record the data update operation of the first server in the first log; log, and perform data synchronization according to the data update operation recorded in the first log during its offline period. Compared with the synchronization method in which all backup data is updated to all data on the second server in the prior art, the data synchronization method described in this solution greatly reduces the amount of data that needs to be updated, shortens the time required for data synchronization, and improves The efficiency of data synchronization further enables the second server to provide data services to users in the shortest possible time after a fault event occurs, reducing user waiting time caused by the server being offline and improving user experience.

The inventors of the present invention have also found that the threads in the existing distributed storage systems usually have the functions of receiving data update requests and searching for data fragments at the same time; the load balancing method of these threads is usually: a certain After a thread receives a set number of requests, it stops receiving requests and generates a handover notification, and the thread that receives the handover notification continues to receive requests; this method can easily lead to a high load on the thread that receives the request first, and then the load on the thread that receives the request is handed over. low, resulting in unbalanced server load.

The inventor of the present invention considers that the process can be divided into a main control process and a work process by function; the main control process is specially responsible for receiving the user's request, and the received request is evenly distributed to a plurality of work processes, and the work process processes the request Response; the method described in this solution can achieve the purpose of greatly improving the load balance of the server and improving the response speed of the server to user requests due to more fully and efficiently utilizing server resources, thereby improving user experience.

Based on the above considerations, the technical solutions of the embodiments of the present invention will be specifically introduced below in conjunction with the accompanying drawings.

The distributed storage system in the embodiment of the present invention includes servers distributed in multiple locations.

Servers at the same location can communicate with each other through a local area network; servers at different locations can communicate with each other through a wide area network.

Each server in the distributed storage system is divided into multiple data shards for storing data; different data shards of the same server have different names; servers with data shards with the same name are sibling servers . Preferably, each server in the distributed storage system in the embodiment of the present invention uses a non-volatile memory such as a hard disk to store data.

For the data to be stored, the distributed storage system can generate a specified number of backups of the data according to the specified number of backups; then, the data is allocated to a data fragment in a server in the distributed storage system for storage; at the same time , allocate the backup of the data to the same-named data fragment of the sibling server of the server for storage; and record the correspondence between the content of the data, the keyword information of the data, and the name of the data fragment storing the data. The keyword information of the data may be summary information of the data; preferably, the distributed storage system may obtain the summary information of the data by using a summary algorithm to perform summary calculation on the content of the data. The digest algorithm can be MD5 (Message Digest algorithm 5, message digest algorithm version 5).

For example, for the data to be stored, the distributed storage system copies a backup of the data according to the specified backup number 2; assigns the data to a certain data fragment of the first server in the distributed storage system for storage; Assign the backup of the data to the same-named data fragment of the second server (that is, the brother server of the first server) for storage; and record the content of the data, the keyword information of the data, and the name of the data fragment that stores the data The corresponding relationship between them is stored in the first server and the second server respectively.

More preferably, the distributed storage system in the embodiment of the present invention includes Leveldb. Leveldb is a relatively efficient open source database developed by Google. Specifically, LevelDB is usually a C/C++ programming language library, which does not contain a network layer itself. The server of the distributed storage system in the embodiment of the present invention includes the leveldb encapsulated with the network layer.

LevelDB usually provides single-process services with high read and write performance; for example, on a 4-core Q6600 CPU machine, data writes per second exceed 400,000TPS (Transaction PerSecond, transaction processing volume per second), while random read performance More than 100000TPS per second.

Taking the first server and the second server as examples, the method for synchronizing data between servers in the distributed storage system according to the embodiment of the present invention will be described in detail below. The schematic flow diagram of the method is shown in Figure 1a, and includes the following steps:

S101: The first server receives the data update request, and searches for data fragments corresponding to the keyword information in the data update request.

Specifically, after receiving the data update request sent by the client, the first server parses out the keyword information; finds and parses out the keyword information from the pre-recorded correspondence between the keyword information and the name of the data slice Corresponding data fragmentation. The data update request may be a data replacement request or a data deletion request.

Preferably, the master control process created in advance by the first server receives the data update requests sent by multiple clients, which may include: the master control process utilizes HTTP (HyperText Transfer Protocol, hypertext transfer protocol) protocol to monitor the network port; utilizes HTTP protocol conversion The format of the monitored data update request, and the converted data update request is used as the received data update request.

The HTTP protocol contains conversion information between multiple data protocols or formats. Compared with the binary protocol, there is no need to add source codes for protocol or format conversion in the main control process and the client. Users can directly browse through the When the server sends a request, the server can directly monitor and receive the request sent by the user through a network port, such as port 8360 of TCP (Transmission Control Protocol), which improves the efficiency of receiving user requests. Moreover, the HTTP protocol has good scalability, and the server of the distributed storage system in the embodiment of the present invention can conveniently expand more functions according to the HTTP protocol.

Based on each received data update request, the main control process generates a working channel for the data update request; and evenly distributes the received data update request and the corresponding working channel to multiple working processes pre-created by the first server. The working channel can be a file descriptor, such as FD in Unix systems. A worker process can handle millions of work channels.

In this embodiment, multiple working processes respectively search for corresponding data fragments according to the keyword information in various assigned data update requests. Since the load among the working processes is relatively average, it is beneficial to use the resources of the server more fully and efficiently, thereby improving the response speed of the server to user requests.

Preferably, the working process parses out the keyword information from the data update request assigned to it; finds out and parses out the key The data fragment corresponding to the word information.

For example, in the schematic diagram of the data synchronization principle shown in Figure 1b, ServerB can be the first server, ServerC can be the second server, and Client is the client; ServerB receives the data update request put query kv3 sent by the Client, and parses out the key Word information; find the data shard named shard3 corresponding to the keyword information.

S102: The first server updates the data fragments according to the update data in the data update request.

Preferably, for each worker thread in the first server, the worker thread parses the update data from the data update request assigned to it, and updates the data determined by the worker thread in step S101 according to the parsed update data. Data sharding.

For example, in the schematic diagram of the data synchronization principle shown in Figure 1b, the first server ServerB parses the content of the updated data kv3 from the received data update request put query kv3, and updates the content of the parsed kv3 to the data fragment shard3 middle.

Preferably, if the data update request received by the worker thread is a data replacement request, the replacement data parsed from the data replacement request will replace the corresponding data in the data slice determined by the worker thread in step S101.

Preferably, if the data update request received by the working thread is a data deletion request, then according to the data to be deleted parsed from the data replacement request, the corresponding data in the data slice determined by the working thread in step S101 is deleted.

More preferably, after the data update is completed, the worker thread returns a message of successful data update to the client through its assigned work channel.

S103: If the first server detects that the second server configured with another data fragment with the same name as the data fragment determined in step S101 is offline, record the update operation of updating data in the first log of the first server.

Specifically, the first server uses the heartbeat thread to detect whether the second server is offline for the second server that is configured with other data fragments with the same name as the data fragment determined in the above step S101: if so, the update in the above step S102 The data update operation is recorded in the first log of the first server.

Preferably, the first log may be a format log. The update operation corresponding to the update data in the first log also records the update data and its keyword information.

Preferably, when the first server detects that the second server is offline, it records the update operations of the plurality of update data of the first server into the first log of the first server in chronological order.

For example, in the schematic diagram of the data synchronization principle shown in Figure 1b, when the first server ServerB detects that the second server ServerC is offline, the replacement operation of replacing the data kv3 in the data fragment shard3 is recorded in the first log, that is, ServerB In the binlog restore.

S104: The second server sends a log synchronization request to the first server.

Specifically, when the second server changes from an offline state to an online state, a log synchronization request is sent to the first server. Preferably, after the second server goes online, use the heartbeat thread in the second server to send a log synchronization request to the first server.

Or, when the second server is in the online state, it periodically sends a log synchronization request to the first server.

S105: The first server returns the first log to the second server in response to the log synchronization request sent by the second server.

Specifically, after receiving the log synchronization request sent by the second server, the first server returns the first log stored in the server to the second server.

S106: The second server performs a data synchronization operation according to the received first log returned by the first server.

Specifically, the second server compares the received first log returned by the first server with the second log prestored in the second server to determine the update operation list.

The second server updates the corresponding local data fragments according to the determined update operation list.

Preferably, the update operation list specifically includes a plurality of update operations for updating data.

The second server may sequentially update corresponding local data fragments according to the chronological sequence of multiple update operations in the update operation list, and the specific method will be described in detail later.

In the above step S106, the specific method for the second server to sequentially update the corresponding local data fragments according to the time sequence of multiple update operations in the update operation list, the flow chart of which is shown in Figure 2a, includes the following steps:

S201: Based on each update operation in the update operation list of the first log, the second server determines keyword information recorded corresponding to the update operation.

Specifically, based on each update operation in the update operation list determined in the above step S106, the second server determines the keyword information recorded corresponding to the update operation from the received first log.

S202: The second server carries the determined keyword information in a data synchronization request and sends it to the first server, and searches for local data fragments corresponding to the determined keyword information.

Specifically, the second server sends the keyword information determined in step S202 to the first server in the data synchronization request.

The second server finds out the data fragments corresponding to the keyword information determined in the above step S202 from the correspondence between the keyword information and the names of the data fragments pre-stored in the server.

For example, in the schematic diagram of the principle of data synchronization shown in FIG. 1b, the second server ServerC finds the data shard shard3 corresponding to the keyword information in ServerC.

S203: The first server receives the data synchronization request sent by the second server, and parses keyword information from it; searches for a local data fragment corresponding to the parsed keyword information.

Specifically, the first server receives the data synchronization request sent by the second server, and parses out the keyword information determined in step S201; , find out the data fragments corresponding to the parsed keyword information.

For example, in the schematic diagram of the principle of data synchronization shown in Figure 1b, the first server ServerB parses the keyword information from the received data synchronization request sent by the second server ServerC, and finds out the information corresponding to the keyword information in the first server ServerB. The corresponding data shard shard3.

S204: The first server returns the found data in the data fragments to the second server.

Specifically, the first server obtains the data corresponding to the keyword information determined in the above step S201 from the data fragment found in the above step S203, and returns the obtained data to the second server.

For example, in the schematic diagram of the principle of data synchronization shown in FIG. 1b, the first server ServerB acquires data kv3 corresponding to keyword information from the data shard3 of ServerB, and returns the acquired kv3 to the second server ServerC.

S205: The second server updates the found local data fragments according to the received data returned by the first server.

Specifically, the second server updates the received data returned by the first server into the data fragments of the second server found in the above step 202 according to the update operation based on the above step S201.

Preferably, if the update operation is a replacement operation, the second server will replace the corresponding data in the second server data fragment found in step 202 with the received data returned by the first server.

For example, in the schematic diagram of the principle of data synchronization shown in FIG. 1b, the second server ServerC replaces the corresponding kv3 in the data shard Shard3 in ServerC with the received data kv3 returned by the first server ServerB.

In fact, the first server in the embodiment of the present invention can also have the same function as the second server in the method flow steps shown in Figure 1a and Figure 2a; the second server can also have the same function as Figure 1a and Figure 2a The first server in the method flow steps shown in Fig. 2a has the same function.

In fact, the distributed storage system of the embodiment of the present invention includes a response method to the data acquisition request sent by the user in the above-mentioned method flow steps shown in FIG. 1a and FIG. 2 .

For example, in the schematic diagram of the data acquisition principle shown in Figure 2b, after ServerA in the distributed storage system receives the data acquisition request get query kv3 sent by the user through the client client, it parses out the keyword information, but pre-stored in ServerA In the corresponding relationship between the keyword information and the name of the data fragment, the parsed keyword information cannot be found; ServerA uses the temporary redirection function in the HTTP protocol to send a temporary redirection with the network address of ServerB to the client. Orientation instructions. The network address may be a URL (Uniform Resource Location, Uniform Resource Locator).

The client sends a data acquisition request to ServerB according to the temporary redirection instruction.

ServerB parses the keyword information from the received data acquisition request get query kv3, and finds the data fragment corresponding to the parsed keyword information from the corresponding relationship between the keyword information stored in ServerB and the name of the data fragment shard3; read data kv3 from shard3, and send the read data to the client.

Based on the data synchronization method between servers in the above-mentioned distributed storage system, an embodiment of the present invention provides a first server in the distributed storage system, the framework diagram of its internal structure is shown in Figure 3, including: data fragmentation A determination module 301 , a data update module 302 , an update operation record module 303 and a synchronization request response module 304 .

Wherein, the data fragment determining module 301 is configured to receive a data update request, and search for a data fragment corresponding to the keyword information in the data update request.

The data update module 302 is configured to update data fragments according to the update data in the data update request.

The update operation recording module 303 is used to record the update operation of the update data into the first log of the first server if it is detected that the second server that is configured with other data fragments with the same name as the data fragment is offline, wherein the first The server and the second server are sibling servers to each other, and both of them are provided with data fragments with the same name.

The synchronization request response module 304 is configured to return the first log to the second server in response to the log synchronization request sent by the second server.

Preferably, the synchronization request response module 304 is also configured to receive the data synchronization request sent by the second server, and parse out keyword information therefrom; search for local data fragments corresponding to the parsed keyword information; divide the found data into The data in the slice is returned to the second server.

Preferably, a schematic diagram of the internal structure of the data fragmentation determination module 301 is shown in FIG. 4 , which may include: a data update request receiving unit 401 and a data fragmentation determination unit 402 .

Wherein, the data update request receiving unit 401 is used for receiving the data update requests sent by multiple clients by the pre-created main control process; based on each received data update request, generating the working channel of the data update request; the main control process Evenly distribute the received data update requests and corresponding working channels to multiple working processes pre-created by the first server.

Specifically, the data update request receiving unit 401 uses the HTTP protocol to monitor the network port by the main control process; uses the HTTP protocol to convert the format of the monitored data update request, and uses the converted data update request as the received data update request.

The data fragment determination unit 402 is used for the work process to search for corresponding data fragments according to the key information in the assigned data update request.

More preferably, as shown in FIG. 3 , the first server further includes: a synchronization request sending module 305 and a data synchronization module 306 .

Wherein, the synchronization request sending module 305 is configured to send a log synchronization request to the second server.

The data synchronization module 306 is configured to perform a data synchronization operation according to the received second log returned by the second server.

The above-mentioned data fragmentation determination module 301, data update module 302, update operation record module 303, synchronization request response module 304, synchronization request sending module 305, data synchronization module 306, and the data update request receiving unit in the data fragmentation determination module 301 For the implementation method of the functions of 401 and the data fragment determination unit 402, please refer to the specific content of the above-mentioned method flow steps shown in FIG. 1a and FIG. 2a, and details will not be repeated here.

Based on the data synchronization method between servers in the above-mentioned distributed storage system, an embodiment of the present invention provides a second server in the distributed storage system, the framework diagram of its internal structure is shown in Figure 3, including: synchronization request sending module 311 and data synchronization module 312.

Wherein, the synchronization request sending module 311 is configured to send a log synchronization request to the first server.

Specifically, the synchronization request sending module 311 periodically sends a log synchronization request to the first server when the second server is online; or sends a log synchronization request to the first server when the second server changes from offline to online.

The data synchronization module 312 is used to perform a data synchronization operation according to the received first log returned by the first server; wherein, the first server and the second server are sibling servers, and both of them are configured with data fragments with the same name.

More preferably, a schematic diagram of the internal structure of the data synchronization module 312 is shown in FIG. 5 , which specifically includes: an update operation list determination unit 501 and a data update unit 502 .

Wherein, the update operation list determining unit 501 is configured to compare the received first log returned by the first server with the local second log to determine the update operation list.

The data update unit 502 is configured to update the corresponding local data fragments according to the update operation list.

Specifically, when the update operation list specifically includes multiple update data, the update operation data update unit 502 sequentially updates the corresponding local data fragments according to the time sequence of the multiple update operations in the update operation list.

Further, the data update unit 502 is further configured to determine the keyword information recorded corresponding to the update operation based on the first log; carry the determined keyword information in the data synchronization request and send it to the first server; search for local and keyword information The data fragment corresponding to the information; according to the received data returned by the first server, update the found local data fragment.

More preferably, as shown in FIG. 3 , the second server in the embodiment of the present invention further includes: a data fragment determination module 313 , a data update module 314 , an update operation record module 315 and a synchronization request response module 316 .

Wherein, the data fragment determining module 313 is configured to receive the data update request, and search for the data fragment corresponding to the keyword information in the data update request.

The data update module 314 is configured to update the data fragments according to the update data in the data update request.

The update operation recording module 315 is used to record the update operation of updating data into the second log of the second server if it is detected that the first server that is configured with other data fragments with the same name as the data fragment is offline, wherein the first The server and the second server are sibling servers to each other, and both of them are provided with data fragments with the same name.

The synchronization request response module 316 is configured to return the second log to the first server in response to the log synchronization request sent by the first server.

Preferably, the synchronization request response module 316 is also used to receive the data synchronization request sent by the first server, and parse out the keyword information therefrom; search for local data fragments corresponding to the parsed keyword information; divide the found data into The data in the slice is returned to the first server.

The above-mentioned synchronization request sending module 311, data synchronization module 312, data fragment determination module 313, data update module 314, update operation record module 315 and synchronization request response module 316, as well as the update operation list determination unit 501 and For the implementation method of the function of the data update unit 502, please refer to the specific content of the steps of the above-mentioned method as shown in FIG. 1a and FIG. 2a, and details will not be repeated here.

In the technical solution of the embodiment of the present invention, the first server of the distributed storage system records the data update operation during the offline period of the second server into a log, and after the second server goes online, compares the local data with the first server's data according to the log. Data synchronization; compared with the data synchronization method of updating all data in the second server with all backup data in the prior art, the data synchronization method of this program greatly reduces the amount of data that needs to be synchronized by the second server and shortens the time for data synchronization. The time required to improve the efficiency of data synchronization, and further, after the failure event occurs, the second server can provide data services for users in the shortest possible time, which can reduce the user waiting time caused by the server being offline, and improve user experience .

Moreover, in the technical solution of the embodiment of the present invention, a main control process and a work process are created in the server; the main control process is specially responsible for receiving the user's request, and evenly distributes the received request to a plurality of work processes, and the work process processes the request Response; the method described in this solution can achieve the purpose of greatly improving the load balance of the server and improving the response speed of the server to user requests due to more fully and efficiently utilizing server resources, thereby improving user experience.

Further, in the technical solution of the embodiment of the present invention, there is no need to add source codes for protocol or format conversion in the process of the server and the client, and the process can directly use the HTTP protocol to interact with the client, which improves the interaction between receiving user requests and the like. operational efficiency.

Those skilled in the art will appreciate that the present invention includes devices related to performing one or more of the operations described in this application. These devices may be specially designed and fabricated for the required purposes, or they may include known devices found in general purpose computers. These devices have computer programs stored therein that are selectively activated or reconfigured. Such a computer program can be stored in a device (e.g., computer) readable medium, including but not limited to any type of medium suitable for storing electronic instructions and respectively coupled to a bus. Types of disks (including floppy disks, hard disks, CDs, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory, read-only memory), RAM (Random Access Memory, random memory), EPROM (Erasable Programmable Read-Only Memory , Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory, Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, a readable medium includes any medium that stores or transmits information in a form readable by a device (eg, a computer).

Those skilled in the art will understand that computer program instructions can be used to implement each block in these structural diagrams and/or block diagrams and/or flow diagrams and combinations of blocks in these structural diagrams and/or block diagrams and/or flow diagrams . Those skilled in the art can understand that these computer program instructions can be provided to general-purpose computers, professional computers, or processors of other programmable data processing methods for implementation, so that the computer or processors of other programmable data processing methods can execute the present invention. A scheme specified in a block or blocks of a structure diagram and/or a block diagram and/or a flow diagram of the invention disclosure.

Those skilled in the art can understand that the various operations, methods, and steps, measures, and solutions in the processes discussed in the present invention can be replaced, changed, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes that have been discussed in the present invention may also be replaced, changed, rearranged, decomposed, combined, or deleted. Further, steps, measures, and schemes in the prior art that have operations, methods, and processes disclosed in the present invention can also be alternated, changed, rearranged, decomposed, combined, or deleted.

The above descriptions are only part of the embodiments of the present invention. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the principles of the present invention. It should be regarded as the protection scope of the present invention.

Claims (10)

1. the method for data synchronization of first server end in distributed memory system, is characterized in that, comprising:

First server receives Data Update request, searches the data fragmentation corresponding with the keyword message in described Data Update request;

According to the more new data in described Data Update request, upgrade described data fragmentation;

The second server off-line of other data fragmentations identical with described data fragmentation title is provided with if detect, the renewal rewards theory of described more new data is recorded in the first daily record of described first server, wherein, described first server and second server be fraternal server each other, is wherein provided with the data fragmentation of same names;

In response to the daily record synchronization request that described second server sends, return the first daily record to second server.

2. method according to claim 1, is characterized in that, described to after second server returns the first daily record, also comprises:

Receive the data synchronization request that second server sends, therefrom parse keyword message;

Search the local data fragmentation corresponding to the keyword message parsed;

Data in the data fragmentation found are back to second server.

3. method according to claim 1 and 2, is characterized in that, described first server receives Data Update request, specifically comprises:

The master control process that described first server is pre-created receives the Data Update request that multiple client sends;

Based on each Data Update request received, generate the working channel of this Data Update request;

Described master control process by the Data Update request that receives and corresponding working channel, multiple progresses of work that uniform distribution is pre-created to described first server;

The described progress of work, according to the keyword message in the Data Update request be assigned to, searches corresponding data fragmentation.

4. the method for data synchronization of second server end in distributed memory system, is characterized in that, comprising:

Daily record synchronization request is sent to first server;

According to the first daily record that the described first server received returns, carry out data syn-chronization operation, wherein, described first server and second server be fraternal server each other, is wherein provided with the data fragmentation of same names.

5. method according to claim 4, is characterized in that, sends daily record synchronization request, comprising to first server:

When described second server is under upper line states, periodically send daily record synchronization request to first server; Or

When described second server changes to presence from off-line state, send daily record synchronization request to first server.

6. the method according to claim 4 or 5, is characterized in that, according to the first daily record that the described first server received returns, carries out data syn-chronization operation, specifically comprises:

The first daily record that the first server received is returned compared with local second daily record, to determine renewal rewards theory list;

According to described renewal rewards theory list, upgrade local corresponding data fragmentation.

7. the first server in distributed memory system, is characterized in that, comprising:

Data fragmentation determination module, for receiving Data Update request, searches the data fragmentation corresponding with the keyword message in described Data Update request;

Data update module, for according to the more new data in described Data Update request, upgrades described data fragmentation;

Renewal rewards theory logging modle, if for the second server off-line being provided with other data fragmentations identical with described data fragmentation title being detected, the renewal rewards theory of described more new data is recorded in the first daily record of described first server, wherein, described first server and second server be fraternal server each other, and it is provided with the data fragmentation of same names;

Synchronization request respond module, for the daily record synchronization request sent in response to described second server, returns the first daily record to second server.

8. first server according to claim 7, is characterized in that,

Synchronization request respond module also for receiving the data synchronization request that second server sends, therefrom parses keyword message; Search the local data fragmentation corresponding to the keyword message parsed; Data in the data fragmentation found are back to second server.

9. the second server in distributed memory system, is characterized in that, comprising:

Synchronization request sending module, for sending daily record synchronization request to first server;

Data simultaneous module, for the first daily record returned according to the described first server received, carries out data syn-chronization operation; Wherein, described first server and second server be fraternal server each other, and it is provided with the data fragmentation of same names.

10. second server according to claim 9, is characterized in that,

Synchronization request sending module, specifically for when described second server is under upper line states, periodically sends daily record synchronization request to first server; Or when described second server changes to presence from off-line state, send daily record synchronization request to first server.