CN115361435B

CN115361435B - Data processing method

Info

Publication number: CN115361435B
Application number: CN202210821136.3A
Authority: CN
Inventors: 王梦杰; 黄敏飞
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-07-13
Filing date: 2022-07-13
Publication date: 2024-07-30
Anticipated expiration: 2042-07-13
Also published as: CN115361435A

Abstract

The embodiment of the specification provides a data processing method, which is applied to a data processing system, wherein the system comprises a client management module, an initial client service module and an updated client service module, and the method comprises the following steps: the client management module is used for responding to the update request, acquiring the update client service module and sending the received data processing request to the update client service module, wherein the data processing request carries data to be processed; the updating client service module acquires unit information of the target data processing unit from the initial client service module under the condition that no target data processing unit for processing the data to be processed exists, and calls the target data processing unit to process the data to be processed based on the unit information, so that data processing efficiency and user experience are improved in the process of upgrading the client.

Description

Data processing method

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to a data processing method.

Background

With the development of computer technology, many internet enterprises will issue clients of specific products to users so that the users use the corresponding products through the clients, for example, clients of a distributed system may be issued to users so that the users can use services provided by the distributed system based on the clients.

However, in the process of upgrading the client, because a long time is required between the new client and the old client for data synchronization operation and restarting operation, the client cannot process data in the upgrading process, so that the data processing efficiency is reduced, and the use experience of a user is poor.

Disclosure of Invention

In view of this, the present embodiments provide a data processing method. One or more embodiments of the present specification are also directed to a data processing system, a computing device, a computer readable storage medium, and a computer program to address the deficiencies of the prior art.

According to a first aspect of embodiments of the present specification, there is provided a data processing method applied to a data processing system, the system including a client management module, an initial client service module, and an update client service module, the method including:

the client management module is used for responding to the update request, acquiring the update client service module and sending the received data processing request to the update client service module, wherein the data processing request carries data to be processed;

And the updating client service module acquires unit information of the target data processing unit from the initial client service module and calls the target data processing unit to process the data to be processed based on the unit information under the condition that the target data processing unit for processing the data to be processed is determined to be absent.

According to a second aspect of embodiments of the present specification, there is provided a data processing system comprising a client management module, an initial client service module, an update client service module, wherein,

The client management module is configured to respond to an update request, acquire the update client service module and send a received data processing request to the update client service module, wherein the data processing request carries data to be processed;

The updating client service module is configured to acquire unit information of a target data processing unit from the initial client service module and call the target data processing unit to process the data to be processed based on the unit information when no target data processing unit for processing the data to be processed is determined.

According to a third aspect of embodiments of the present specification, there is provided a computing device comprising:

A memory and a processor;

The memory is configured to store computer executable instructions that, when executed by the processor, implement the steps of the data processing method described above as being applied to the client management module, the initial client service module, or the update client service module.

According to a fourth aspect of embodiments of the present specification, there is provided a computer readable storage medium storing computer executable instructions which, when executed by a processor, implement the steps of the data processing method as described above applied to a client management module, an initial client service module or an update client service module.

According to a fifth aspect of embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method described above as applied to a client management module, an initial client service module or an update client service module.

The data processing method provided by the specification is applied to a data processing system, the system comprises a client management module, an initial client service module and an update client service module, and the method comprises the following steps: the client management module is used for responding to the update request, acquiring the update client service module and sending the received data processing request to the update client service module, wherein the data processing request carries data to be processed; and the updating client service module acquires unit information of the target data processing unit from the initial client service module and calls the target data processing unit to process the data to be processed based on the unit information under the condition that the target data processing unit for processing the data to be processed is determined to be absent.

The method comprises the steps of firstly acquiring an update client service module corresponding to an initial client service module through a client management module, upgrading the initial client service module based on the update client service module, and then sending a received data processing request to the update client service module for processing.

Drawings

Fig. 1 is a schematic application scenario diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram illustrating a data processing method applied in an online hot upgrade scenario for a client according to an embodiment of the present disclosure;

FIG. 3 is a flow chart of a method of data processing provided in one embodiment of the present disclosure;

FIG. 4 is a process flow diagram of a data processing method according to one embodiment of the present disclosure;

FIG. 5 is a schematic diagram of a data processing system according to one embodiment of the present disclosure;

FIG. 6 is a block diagram of a computing device provided in one embodiment of the present description.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many other forms than described herein and similarly generalized by those skilled in the art to whom this disclosure pertains without departing from the spirit of the disclosure and, therefore, this disclosure is not limited by the specific implementations disclosed below.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "in response to a determination" depending on the context.

First, terms related to one or more embodiments of the present specification will be explained.

Thermal upgrade: and upgrading the service without stopping.

File stream: refers to a file stream class (FILESTREAM class), a class used to implement operations such as reading data from a file and writing data to a file.

Dynamic library: generally referred to as a dynamic link library.

Symbol Versioning: symbol version mechanism, a technique for version control of dynamic library symbols.

With the development of computer technology, many internet enterprises will issue clients of a specific product to users so that the users use the corresponding product through the clients, for example, clients of a distributed system may be issued to users so that the users can use the distributed system based on the clients.

However, the client of the distributed system is in the process of the user, once the client is released to the user, the developer cannot control the version of the client, and the developer can only push the user to upgrade the version of the client manually, so that a large amount of manpower resources are consumed, and the user is also burdened.

In view of this problem, the present specification provides two client upgrade schemes: the scheme of a customer end upgrade is, export the customer end in the form of dynamic library; when the client has a new version, the new client dynamic library is downloaded to the machine where the client is located, so that when the user process is restarted, the new client version can be used. However, the problem of this solution is that the process still needs to be restarted by the user, but for some users, the process normally runs for a long time, and if the process needs to be restarted, data/task migration needs to be performed, which may bring additional burden to the process and may affect the service provided by the client to the user.

Another approach to online hot-upgrade of clients is to separate the client from the user process as a separate service process. When a client is hot-upgraded, a new version is used to start the client process (when the old client process is still in service), and then the file stream in the old client is reopened in the new client and service is resumed. However, this solution has the disadvantage that in some scenarios the file stream is not allowed to be re-opened for reasons of hot upgrade, or the client is not allowed to exist as a separate process. In addition, since data is transferred from the user process to the client process through inter-process data transfer, additional overhead is brought, and delay and throughput are affected.

Based on this, in the present specification, a data processing method is provided, and the present specification relates to a data processing system, a computing device, a computer-readable storage medium, and a computer program, which are described in detail in the following embodiments one by one.

Fig. 1 shows an application scenario of a data processing method according to an embodiment of the present disclosure, where the data processing method is applied to a data processing system, and the system includes a client management module 102, an initial client service module 106, and an update client service module 104, and the method includes:

the client management module 102 responds to an update request, acquires the update client service module 104, and sends a received data processing request to the update client service module 104, wherein the data processing request carries data to be processed;

the update client service module 104 obtains unit information of a target data processing unit from the initial client service module 106 and calls the target data processing unit to process the data to be processed based on the unit information when it is determined that the target data processing unit to process the data to be processed does not exist.

The client service module may be understood as a client, and is a client program running on the user terminal, or a client instance running on the user terminal; in order to avoid excessive redundancy, the client service module is referred to as a client in the following embodiments; correspondingly, the update client service module 104 may be referred to as an update client, and the initial client service module 106 may be referred to as an initial client.

The data processing system may be understood as a system that processes data, such as a distributed storage system, a distributed file system, and the like. In practical applications, the modules of the client management module 102, the initial client service module 106, the update client service module 104, etc. included in the data processing system may be run in a user terminal. The client management module 102 may be understood as a module for managing client service modules in the data processing system. In practical application, in the data processing method provided in the present specification, a client management component is newly added in the distributed file system, where the client management component can be understood as the client management module 102, and the client management component can be responsible for forwarding a request to a client and managing a client thermal upgrade; it should be noted that the client management component (i.e., the client management module 102) may be a client management instance, a client management program, etc. for managing clients, which is not specifically limited in this specification. In addition, the logic of the client management component is simpler, and generally no upgrade is needed. In addition, the client management component is in the process of the user, and the client management component can load the client dynamic library by itself without the user care.

The update client service module 104 may be understood as a new client that performs a hot upgrade to an old version client in the distributed system; it should be noted that, the client issues externally in the form of a dynamic library.

The initial client service module 106 may be understood as a client of an older version in the distributed system. The update request may be understood as a request that instructs the client management model to upgrade the initial client service module 106. The data processing request may be understood as a request requiring the client to process, e.g., a data read request, a data write request, etc. The data to be processed can be understood as data which needs to be processed by the client; for example, data that a client needs to perform a write operation, data that a client needs to perform a read operation, and so on, the data processing unit may be understood as a class that performs various types of operations on data, such as a filestream class. The target data processing unit may be understood as a class that performs operations on the data to be processed. The unit information may be understood as file stream information of the data processing unit. The filestream information may be metadata, for example, metadata of a filestream class.

Specifically, the data processing method provided in the present disclosure provides an online hot upgrade scheme of a distributed file system client, where a client management module 102 in the data processing system can receive an update request for updating an initial client service module 106 in the data processing system, and the client management module 102 can obtain the updated client service module 104 by responding to the update request, so as to upgrade the initial client service module 106 based on the updated client service module 104. Meanwhile, in the process of hot upgrading, in order to ensure that the client service module can provide services for users, the client management model timely processes the received data processing request, and after the client management model sends the received data processing request carrying the data to be processed to the update client service module 104, the update client service module 104 firstly determines whether a target data processing unit for processing the data to be processed exists.

If not, that is, if there is no target data processing unit for processing the data to be processed, the update client service module 104 can obtain the unit information of the target data processing unit from the initial client service module 106, and call the target data processing unit to process the data to be processed based on the unit information.

If so, that is, if there is a target data processing unit for processing the data to be processed, the update client service module 104 directly invokes the target data processing unit to process the data to be processed.

The data processing method provided in the present specification will be described below by taking an application in an online hot upgrade scenario of a distributed file system client as an example. Referring to fig. 2, fig. 2 is a schematic processing diagram of a data processing method applied to an online hot upgrade scenario for a client according to an embodiment of the present disclosure. The client management module is a client management component, the initial client service module is a client dynamic library (version 1), the updated client service module is a client dynamic library (version 2), and the unit information is file stream information. It should be noted that, the client management component forwards the data writing request sent by the user to the client currently running.

Based on the above, when a new version of client is generated, the central management node generates an update instruction for the client dynamic library (version 1) to the client management component, wherein the update instruction carries identification information of the client dynamic library (version 2); and after receiving the update instruction sent by the central management node, the client management component running in the user process downloads the new version of the client dynamic library (version 2) from the central management node to the local, which can be understood as a local disk, based on the identification information.

It should be noted that, the client management component running in the user process maintains two client instances, one as a main instance, i.e. the instance currently being served, and the other as a standby instance. The client instance is not in the running user process.

Based on this, after the client management component downloads the new version of the client dynamic library (version 2), the old version of the client (client dynamic library (version 1)) is thermally upgraded based on the new version of the client (i.e., the client dynamic library (version 2)), and during the process of performing the thermal upgrade, the client management component first destructs the local standby instance (if there is a standby instance), then loads the new client dynamic library, re-instantiates the new dynamic library (i.e., the new version of the client), obtains the standby client instance generated by the new version of the client, and initializes the standby client instance.

After the initialization of the standby client instance (i.e., the new version client instance) corresponding to the new version client is completed, the new version client instance exchanges and synchronizes information with the old client instance, and file stream information is obtained from the old version client instance, that is, file stream information migration between the client dynamic library (version 1) and the client dynamic library (version 2) in fig. 2; the filestream information may be metadata of a filestream class. It should be noted that, the client side dynamic library (version 2) may be provided by calling an information synchronization function B provided by the client side dynamic library (version 1), where the information synchronization function B is a function for obtaining file stream information that is not yet synchronized to the new client side in the old client side, and synchronizing the file stream information from the client side dynamic library (version 1) by calling the information synchronization function B. After the client dynamic library (version 1) and the client dynamic library (version 2) complete file stream information synchronization, it can be determined that the client dynamic library (version 1) is hot-upgraded.

Meanwhile, in order to ensure that the data processing method provided in the present disclosure can process the request sent by the user during the client hot upgrade process, and ensure that the user does not perceive the client upgrade, the client management component forwards the received data writing request to the new version client instance, that is, the client dynamic library (version 2) in fig. 2.

After receiving the data writing request, the new version client instance determines the data to be written carried in the data writing request and the file identifier A1 to which the data to be written needs to be written.

Then, the new version client instance determines file stream information corresponding to the file identifier A1 from file stream information held by the new version client instance based on the file identifier A1, calls the file stream class A2 based on the file stream information, instantiates the file stream class A2, and obtains an instantiated file stream A3; based on the file stream A3, the data to be written is written into the file corresponding to the file identifier A1. It should be noted that, there is a correspondence between the file stream class and the files, and one file stream class may correspond to one or more files. Data may be written to or read from a corresponding file through a filestream class.

However, in the new version client instance, if the file stream information corresponding to the file identifier A1 does not exist, it is stated that the file stream information is not synchronized from the old version client instance, so, in order to avoid the problem that the data writing request cannot be processed in time due to the fact that the new version client does not synchronize the file stream information in time, the new version client instance can also call an information acquisition function a provided by the old version client instance, where the information acquisition function a is a function of acquiring the specified file stream information from the old version client instance, and by calling the information acquisition function a, the new version client instance can acquire the file stream information corresponding to the file identifier A1 from the old version client instance.

Then, the new version client instance can call the file stream class A2 based on the file stream information, and instantiate the file stream class A2 to obtain an instantiated file stream A3; based on the file stream A3, the data to be written is written into the file corresponding to the file identifier A1. Therefore, in the process of carrying out hot upgrading on the client, the request of the user can be processed, and the user is ensured to have no perception.

In the data processing method provided by the specification, an online hot upgrading mode of the distributed file system client is provided, so that a maintainer (namely a client management component) of the back-end distributed file system can perform hot upgrading by himself without user process perception, and meanwhile, the upgrading has no influence on the service of a user, and the user is ensured to have no perception.

FIG. 3 shows a flow chart of a data processing method according to one embodiment of the present disclosure, where the data processing method is applied to a data processing system, and the data processing system includes a client management module, an initial client service module, and an update client service module.

Step 302: the client management module responds to the update request, acquires the update client service module, and sends the received data processing request to the update client service module.

Wherein the data processing request carries data to be processed.

In an embodiment provided in the present disclosure, when determining that a new version of a client is available, the central management node instructs the client management module to actively perform a hot upgrade on the client, so as to ensure that the developer can control the version of the client, and avoid the consumption of labor cost caused by manually pushing a user to perform an upgrade, and specifically, the client management module, in response to an update request, obtains the update client service module, including:

the client management module receives an update request which is sent by a client management node and aims at the initial client service module, wherein the update request carries a module identifier of the update client service module;

And acquiring the updated client service module from the client management node based on the module identifier.

The module identifier may be understood as an identifier that uniquely identifies a client service module, that is, an identifier that uniquely identifies a client, for example, a version number of the client, a client number, a client name, and the like, which is not specifically limited in this specification.

The client management node can be understood as a node which stores the latest version of the client and instructs the client management module to upgrade the client; the client management node may be a node in a distributed storage system; for example, the central management node in the above embodiments, that is, the data processing system includes a client management node.

Specifically, the client management node generates an update request for the initial client service module based on the module identifier of the update client service module when determining that the client has a new version; and sending the update request to a client management module;

After receiving the update request, the client management module can acquire the module identifier of the update client service module carried in the update request, generate a client downloading request based on the module identifier of the update client service module, and send the client downloading request to the client management node.

And the client management node searches a corresponding update client service module based on the module identifier of the update client service module carried in the client downloading request in the process of receiving the client downloading request, and sends the update client service module to the client management module. Based on this, the client management module obtains the update client.

In an embodiment provided in the present specification, the client management node is not only responsible for management of client thermal upgrade, but also capable of forwarding a request to a client for processing, so as to enable the client service module to process a user request, specifically, sending a received data processing request to the update client service module, including:

the client management module receives a data processing request sent by a user and sends the data processing request to the updating client service module for processing.

Specifically, in the case where the user needs to process the data, for example, read the data, write the data, etc., the user sends a data processing request to a client management module running in the user process, and the client management module, upon receiving the data processing request, forwards the data processing request to the currently running update client service module.

It should be noted that, after the client management module obtains the update client service module, the update client service module is determined as the currently running client service module, so as to send the received data processing request to the update client service module.

In an embodiment provided in the present disclosure, the client management module may maintain two client instances, a main client and a corresponding backup client (i.e. a backup client instance), so as to ensure data security, and in case that an error occurs in a client, the backup client may be used to replace the main client to work, so as to ensure that the client stably provides services to a user, based on this, in a process of acquiring an update client service module, the backup client service module corresponding to the initial client service module may be deleted, and then the update client service module may be reloaded, and specifically, before the received data processing request is sent to the update client service module, the method further includes:

the client management module determines a backup client service module corresponding to the initial client service module;

And deleting the backup client service module and loading the update client service module through a version control mechanism.

In practical applications, the client management module maintains two client instances, one as a primary client instance, i.e., the instance currently being serviced, and the other as a standby client instance. Therefore, when the hot upgrade is performed, the client management module first destructs (i.e. deletes) the local standby client instance, and then loads the new client dynamic library to generate the new version client instance. Based on this, the initial client service module may be understood as the master client instance of the old version client maintained by the client management module; the backup client service module may be understood as a backup client instance of an old version client maintained by the client management module.

The version control mechanism may be any mechanism for avoiding symbol collision of two versions of client dynamic libraries, and this specification is not limited in particular, for example, the version control mechanism may be Symbol Versioning. In practical application, in order to avoid the problem that symbol conflicts are generated in the dynamic library of the client of two versions, the symbols of the dynamic library can be controlled in a Symbol Versioning mode, and the symbols of the clients of different versions are different.

Specifically, after obtaining the updated client service module, the client management module determines an initial client service module maintained by itself and a backup client service module corresponding to the initial client service module; and then deleting the backup client service module, loading the update client service module through a version control mechanism, and facilitating subsequent hot upgrading of the initial client service module based on the update client service module.

Step 304: and the updating client service module acquires unit information of the target data processing unit from the initial client service module and calls the target data processing unit to process the data to be processed based on the unit information under the condition that the target data processing unit for processing the data to be processed is determined to be absent.

Specifically, in the case that the update client service module determines that there is no target data processing unit for processing data to be processed, the update client service module can acquire unit information of the target data processing unit from the initial client service module and call the target data processing unit based on the unit information, where it is noted that the data processing unit may be stored in a specific storage medium (for example, the data processing unit may be stored in a storage medium such as a local disk, a cache, or a memory of the user terminal), and the update client service module can call the corresponding target data processing unit through the unit information. The update client service module then processes the data to be processed based on the target data processing unit.

Along the above example, the unit information may be file stream information, and the target data processing unit is a file stream class for processing information to be processed; based on the above, the new version client instance can obtain the file stream information corresponding to the file stream class from the old version client instance under the condition that the file stream class for processing the information to be processed does not exist, call the file stream class based on the file stream information, instantiate the file stream type to obtain the instantiated file stream, and read the information to be processed based on the instantiated file stream.

Further, in an embodiment provided in the present specification, the determining that there is no target data processing unit that processes the data to be processed includes:

The updating client service module determines unit information of a data processing unit and attribute information of the data to be processed, wherein the unit information is information acquired by the updating client service module from the initial client service module;

And in the case that the unit information does not match the attribute information, determining that a target data processing unit for processing the data to be processed does not exist.

The attribute information of the data to be processed may be understood as a data identifier corresponding to the data to be processed, and the data identifier may be understood as information that uniquely identifies one data, such as a data name, a data number, and the like. In practical applications, the data to be processed may be a file to be processed, and based on this, attribute information of the data to be processed may be understood as a file identifier corresponding to the file to be processed, where the file identifier may be understood as information that uniquely identifies a file, such as a file name, a file number, and so on.

Specifically, after determining the data to be processed, the update client service module can determine attribute information of the data to be processed, and determine attribute information of each data processing unit from own holding unit information, where the attribute information of the data processing unit can be understood as attribute information of the data to be processed corresponding to the data processing unit.

And matching the attribute information of the data processing unit with the attribute information of the data to be processed, and under the condition that the matching result is inconsistent, determining that the target data processing unit for processing the data to be processed does not exist by the updating client service module.

In one embodiment provided in this specification, to enable information exchange and synchronization between a new client and an old client, a series of functions may be defined in the client and function pointers of the old client may be transferred to the new client for invocation. When the generation and initialization of the new version client instance is completed and the function pointer of the old client is transferred to the new client, the active and standby instances can be exchanged, and the new client instance provides service for the user and receives subsequent requests. When the new version client instance receives the user request, but the new client locally has no file stream information corresponding to the data to be processed carried in the request, a function A provided by the old client and capable of acquiring the specified file stream information can be called to acquire the specified file stream information, so that the new version client can process the request timely. Specifically, the obtaining the target unit information from the initial client service module includes:

And the updating client service module is used for acquiring the target unit information from the initial client service module by calling the information acquisition function provided by the initial client service module.

The information acquiring function may be understood as a function capable of acquiring specified unit information from the initial client service module.

Specifically, the update client service module can obtain the target unit information from the initial client service module by calling the information obtaining function provided by the initial client service module. Further, in an embodiment provided in the present disclosure, the updating the client service module, by calling an information obtaining function provided by the initial client service module, obtains target unit information from the initial client service module, including:

The updating client service module determines an information acquisition function provided by the initial client service module, and sends attribute information of the data to be processed to the initial client service module by calling the information acquisition function;

The initial client service module determines target unit information corresponding to the attribute information from unit information held by the initial client service module and sends the target unit information to the updating client service module;

And the updating client service module receives the target unit information sent by the initial client service module.

Specifically, the update client service module determines a function pointer corresponding to the information acquisition function, where the function pointer is sent to the update client service module by the initial client service module, when it is determined that there is no target data processing unit for processing data to be processed locally.

And then, calling the information acquisition function based on the function pointer, and sending the attribute information of the data to be processed to the initial client service module.

After receiving the attribute information, the initial client service module can determine the unit information held by itself, and determine target unit information corresponding to the attribute information from the unit information, for example, determine a file identifier of a file (i.e. the data to be processed) corresponding to each file stream from the metadata of the file stream, and match the file identifier with the received file identifier. And sending the matched and consistent file stream information to the updating client service module.

Finally, the update client service module receives the target unit information sent by the initial client service module.

Further, in the embodiment provided in the present disclosure, in the process of transmitting unit information between the update client service module and the initial client service module, in order to solve the compatibility problem between the old client and the new client, the old client may sequence the file stream information into a binary stream and send the binary stream to the new client, and the new client deserializes the binary stream, thereby obtaining the file stream information, and avoiding the compatibility problem between the old client and the new client. Specifically, the sending the target unit information to the update client service module includes:

the initial client service module performs serialization processing on the target unit information held by the initial client service module to obtain a binary stream corresponding to the target unit information, and sends the binary stream corresponding to the target unit information to the update client service module;

Correspondingly, the updating the client service module, receiving the target unit information sent by the initial client service module, includes:

and the updating client service module performs deserialization processing on the received binary stream corresponding to the target unit information to obtain the target unit information.

Specifically, after determining the target unit information, the initial client service module performs serialization processing on the target unit information held by itself to obtain a binary stream corresponding to the target unit information. In the process of serializing the target unit information, any mode of serializing the unit information may be adopted, which is not particularly limited in this specification.

And then, the initial client service module sends the binary stream corresponding to the target unit information to the updating client service module.

And after receiving the binary stream, the update client service module performs deserialization processing on the binary stream corresponding to the target unit information, thereby obtaining the target unit information.

In practical applications, in order for a new client to acquire metadata information (i.e., file stream information) about a file stream in an old client so as to facilitate continuous service, two functions need to be defined in the client to synchronize the file stream information, wherein the first function is to acquire specified file stream information, i.e., an information acquisition function; and the second function is a function of acquiring a file stream in the old client that has not been synchronized to the new client, i.e. an information synchronization function. The new client can synchronize file stream information from the old client by calling the information synchronization function, specifically, the update client service module, before acquiring unit information of the target data processing unit from the initial client service module if it is determined that the target data processing unit for processing the data to be processed does not exist, further includes:

and the updating client service module is used for acquiring the unit information of the data processing unit from the initial client service module by calling the information synchronization function provided by the initial client service module.

Specifically, the updating the client service module, by calling the information synchronization function provided by the initial client service module, obtains the unit information of the data processing unit from the initial client service module, including:

The updating client service module determines an information synchronization function provided by the initial client service module and sends an information synchronization request to the initial client service module by calling the information synchronization function;

And the initial client service module is used for responding to the information synchronization request and synchronizing the unit information of the data processing unit to the updating client service module.

Specifically, before the update client service module processes the data to be processed, a function pointer corresponding to the information synchronization function is determined, where the function pointer is sent by the initial client service module to the update client service module.

And then, calling the information synchronization function based on the function pointer, and sending an information synchronization request to the initial client service module through the information synchronization function, wherein the information synchronization request is a request for requesting the initial client service module to send local unit information of the initial client service module to the updated client service module.

And the initial client service module can respond to the information synchronization request to send the self-held unit information to the updating client service module after receiving the information synchronization request.

Finally, the update client service module can obtain the unit information held by the initial client service module. Therefore, the initial client is thermally upgraded, the data to be processed can be processed in the process of upgrading the client, and the data processing efficiency is improved.

Further, in the embodiment provided in the present specification, in order to address the compatibility problem between the old and new clients, the old client may serialize the file stream information into a binary stream and send the binary stream to the new client, where the new client deserializes the binary stream, thereby obtaining the file stream information, and avoiding the compatibility problem between the old and new clients. Specifically, the synchronizing the unit information of the data processing unit to the update client service module includes:

the initial client service module performs serialization processing on unit information of a data processing unit to obtain a binary stream corresponding to the unit information, and sends the binary stream corresponding to the unit information to the update client service module;

And the updating client service module performs deserialization processing on the received binary stream corresponding to the unit information to obtain the unit information of the data processing unit.

Specifically, the initial client service module performs serialization processing on unit information of a data processing unit held by the initial client service module to obtain a binary stream corresponding to the unit information, and sends the binary stream corresponding to the unit information to the update client service module;

and the updating client service module receives the binary stream corresponding to the unit information and then performs deserialization processing on the binary stream to obtain the unit information of the data processing unit.

It should be noted that, in order to ensure the latest of the file stream information, when the old client synchronizes the information of a certain file stream, it is necessary to wait for all the ongoing requests of the file stream to end, and the new client continuously calls the information synchronization function in the background to synchronize other file stream information until all the file stream information is synchronized.

In an embodiment provided in this specification, when the old client is destructed, it is necessary to wait for all file streams to migrate, and the new client will not call the functions of the old client. To this end, an information synchronization completion function is defined in the client to inform the completion of the migration of the filestream, which can be parsed. And when the information synchronization function returns that all file streams are synchronously completed and no ongoing information acquisition function exists, the new client can call the information synchronization completion function of the old client. At this time, the hot upgrade is completed completely, and the client management component may perform a deconstructing on the old client, specifically, the data processing method further includes:

The updating client service module determines the current transmission state of the unit information and the target unit information, and calls the initial client service module to provide an information synchronization completion function and sends an information synchronization completion instruction to the initial client service module under the condition that the current transmission state meets the transmission completion condition;

the initial client service module determines that the information synchronization is completed based on the information synchronization completion instruction and sends a client deletion notification to the client management module;

and the client management module responds to the client deletion notification to delete the initial client service module.

The current transmission state of the unit information may be understood as information indicating whether the unit information is still being synchronously transmitted, and the current transmission state of the target unit information may be understood as information indicating whether the target unit information is being transmitted. The current transmission state satisfying the transmission completion condition can be understood as having completed the information transmission of the unit information as well as the target unit information.

The information synchronization completion instruction may be understood as a function indicating that the initial client data has been synchronously transferred.

Along the above example, in the new version client instance, it is determined that the file stream information has been completed synchronously, and in the case that the file stream information acquired from the old client instance is not needed currently, the information synchronization completion function provided by the old version client is called, and an information synchronization completion instruction is sent to the old version client.

After the old version client instance receives the information synchronization completion instruction, determining that the information synchronization is completed, sending an instance destructing instruction to the client management component, and destructing the old version client instance based on the instance destructing instruction by the client management component.

It should be noted that, after the destructing of the old version client instance is completed, the client management component determines the new version client instance as the master instance, and creates a corresponding standby instance for the new version client instance, so as to ensure the security of the client instance.

According to the data processing method provided by the specification, firstly, the update client service module corresponding to the initial client service module is obtained through the client management module, so that the update client service module is used for upgrading the initial client service module, then a received data processing request is sent to the update client service module for processing, and the update client service module calls the target data processing unit to process the data to be processed through the unit information of the target data processing unit obtained from the initial client service module under the condition that the target data processing unit for processing the data to be processed is determined not to exist, thereby realizing that the data to be processed can be processed in the process of upgrading the client, improving the processing efficiency of the data and improving the use experience of a user.

The following describes, with reference to fig. 4, an example of application of the data processing method provided in the present specification in an online hot upgrade scenario for a client, where the data processing method is further described. Fig. 4 is a flowchart of a processing procedure of a data processing method according to an embodiment of the present disclosure, which specifically includes the following steps.

Step 402: the client management component downloads the new version of the client dynamic library from the central management node.

Step 404: the client management component deletes the standby client of the old version client and loads the new client instance.

Specifically, a client management component running in a user process maintains two client instances, one as a master, i.e., currently serving, instance and the other as a standby.

Based on this, after the client management component downloads the new version of the client dynamic library (version 2), the local standby instance is firstly destructed, then the new client dynamic library is loaded, the new dynamic library (i.e. the new version client) is used for re-instantiating, the standby client instance generated by the new version client is obtained, and the standby client instance is initialized.

Step 406: the old client instance sends its own function pointer to the new client instance.

The function pointer comprises a function A, a function B and a function C.

The function A is a function for acquiring specified file stream information from an old client instance; the function B is a function for acquiring file stream information which is not synchronized to a new client in an old client; function C is a function indicating that the old client instance filestream information has been completed synchronously.

Step 408: the new client instance invokes function B to obtain filestream information from the old client instance.

Specifically, the new client instance calls the function B through a function pointer corresponding to the function B, and sends an information synchronization request to the old client instance;

the old client instance can synchronously transmit the file stream information held by itself to the new client instance after receiving the information synchronization request.

It should be noted that, when synchronizing information of a certain file stream, an old client needs to wait for all ongoing requests of the file stream to end. In order to solve the compatibility problem between the old and new clients, the old client may serialize the file stream information into a binary stream, and the new client may then deserialize.

Step 410: the client management component sends a data writing request which is sent by a user and carries data to be written to the new client instance.

Step 412: the new client instance determines whether there is file stream information for processing the file stream of the data to be written, if so, then step 414 is executed, and if not, then step 406 is executed.

Specifically, the new version client instance determines the data to be written carried in the data writing request, and the file identifier A1 to which the data to be written needs to be written.

Then, the new version client instance determines, based on the file identifier A1, whether there is file stream information corresponding to the file identifier A1 from the file stream information held by itself, if so, step 414 is executed, and if not, step 416 is executed.

Step 414: the new client instance calls the file stream class to process the data to be processed.

Specifically, the new client calls a filestream class corresponding to the filestream information, instantiates the filestream class, and processes data to be processed based on the filestream obtained after the instantiation.

Step 416: the new client instance obtains filestream data from the old client instance based on function a.

Specifically, when a user request arrives, if the new client has no information of the file stream locally, the function A of the old client is called to synchronize, and meanwhile, the request of the file stream is suspended, and after the information stream is synchronized, the suspended requests are sent in sequence.

The method comprises the steps of calling a function A of an old client to synchronize file stream information, wherein the method specifically comprises the following steps:

The new client instance calls the function A based on the function pointer corresponding to the function A, and sends the file identification corresponding to the data to be written to the old client instance;

The old client instance determines the filestream information corresponding to the filemark from itself and sends the filestream information to the new client instance.

Step 418: the new client instance calls the file stream class to process the data to be processed based on the file stream information sent by the old client.

Step 420: after the new client instance determines that the information synchronization is completed, the calling function C informs the old client instance that the information synchronization is completed.

Specifically, the new client instance calls the function C based on the function pointer corresponding to the function C, and sends an information synchronization completion notification to the old client instance.

Step 422: the old client instance determines that the information synchronization is completed based on the information synchronization completion notification and sends an instance destructor to the client management component.

Step 424: the client management component performs a destruct of the old client instance based on the instance destruct instruction.

According to the data processing method provided by the specification, the online non-thermal upgrading of the distributed file system client is realized by loading the client dynamic libraries of two versions simultaneously and synchronizing file stream information of the new client and the old client in the memory. And, utilize Symbol versioning to load the client side dynamic library of two versions at the same time, make both can realize the hot upgrade of the incessant service, avoid the performance loss brought under the double process mode at the same time. And the client dynamic libraries of the two versions synchronize file stream information in the memory, so that the new client and the old client can be in seamless connection, and the influence of hot upgrading on the service delay of the user is minimized.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of a data processing system, and fig. 5 shows a schematic structural diagram of data processing provided in one embodiment of the present disclosure. As shown in fig. 5, the system includes a client management module 502, an initial client service module 506, an update client service module 504, wherein,

The client management module 502 is configured to respond to an update request, acquire the update client service module 504, and send a received data processing request to the update client service module 504, where the data processing request carries data to be processed;

The update client service module 504 is configured to, in a case where it is determined that there is no target data processing unit for processing the data to be processed, acquire unit information of the target data processing unit from the initial client service module 506, and call the target data processing unit to process the data to be processed based on the unit information.

Optionally, the client management module 502 is further configured to:

Receiving an update request for the initial client service module 506 sent by a client management node, wherein the update request carries a module identifier of the update client service module 504;

the update client service module 504 is obtained from the client management node based on the module identification.

Optionally, the client management module 502 is further configured to:

A data processing request sent by a user is received and sent to the update client service module 504 for processing.

Optionally, the client management module 502 is further configured to:

determining a backup client service module corresponding to the initial client service module 506;

the backup client service module is deleted and the update client service module 504 is loaded via a version control mechanism.

Optionally, the update client service module 504 is further configured to:

determining unit information of a data processing unit and attribute information of the data to be processed, wherein the unit information is information acquired by the update client service module 504 from the initial client service module 506;

Optionally, the update client service module 504 is further configured to:

The unit information of the data processing unit is obtained from the initial client service module 506 by calling an information synchronization function provided by the initial client service module 506.

Optionally, the update client service module 504 is further configured to:

Determining an information synchronization function provided by the initial client service module 506, and sending an information synchronization request to the initial client service module 506 by calling the information synchronization function;

The initial client service module 506 synchronizes the unit information of the data processing unit to the update client service module 504 in response to the information synchronization request.

Optionally, the initial client service module 506 is further configured to:

Serializing the unit information of the data processing unit to obtain a binary stream corresponding to the unit information, and sending the binary stream corresponding to the unit information to the update client service module 504;

The update client service module 504 performs deserialization processing on the binary stream corresponding to the received unit information to obtain the unit information of the data processing unit.

Optionally, the update client service module 504 is further configured to:

Target unit information is obtained from the initial client service module 506 by calling an information obtaining function provided by the initial client service module 506.

Optionally, the update client service module 504 is further configured to:

Determining an information acquisition function provided by the initial client service module 506, and sending attribute information of the data to be processed to the initial client service module 506 by calling the information acquisition function;

The initial client service module 506 determines target unit information corresponding to the attribute information from the unit information held by itself, and sends the target unit information to the update client service module 504;

The update client service module 504 receives the target unit information sent by the initial client service module 506.

Optionally, the initial client service module 506 is further configured to:

serializing the target unit information held by the user to obtain a binary stream corresponding to the target unit information, and sending the binary stream corresponding to the target unit information to the update client service module 504;

Accordingly, the update client service module 504 is further configured to:

The update client service module 504 performs deserialization processing on the binary stream corresponding to the received target unit information to obtain target unit information.

Optionally, the update client service module 504 is further configured to:

Determining the current transmission state of the unit information and the target unit information, and calling the initial client service module 506 to provide an information synchronization completion function and sending an information synchronization completion instruction to the initial client service module 506 when the current transmission state meets a transmission completion condition;

The initial client service module 506 is further configured to:

Based on the information synchronization completion instruction, determining that information synchronization is completed, and sending a client deletion notification to the client management module 502;

the client management module 502 is further configured to:

responsive to the client delete notification, the initial client service module 506 is deleted.

According to the data processing system provided by the specification, firstly, the update client service module corresponding to the initial client service module is acquired through the client management module, so that the update client service module is used for upgrading the initial client service module, then a received data processing request is sent to the update client service module for processing, and the update client service module calls the target data processing unit to process the data to be processed through the unit information of the target data processing unit acquired from the initial client service module under the condition that the target data processing unit for processing the data to be processed is determined not to exist, thereby realizing that the data to be processed can be processed in the process of upgrading the client, improving the processing efficiency of the data and improving the use experience of a user.

The foregoing is a schematic illustration of a data processing system of this embodiment. It should be noted that, the technical solution of the data processing system and the technical solution of the data processing method belong to the same conception, and details of the technical solution of the data processing system, which are not described in detail, can be referred to the description of the technical solution of the data processing method.

Fig. 6 illustrates a block diagram of a computing device 600 provided in accordance with one embodiment of the present description. The components of computing device 600 include, but are not limited to, memory 610 and processor 620. The processor 620 is coupled to the memory 610 via a bus 630 and a database 650 is used to hold data.

Computing device 600 also includes access device 640, access device 640 enabling computing device 600 to communicate via one or more networks 660. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 640 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.

In one embodiment of the present description, the above-described components of computing device 600, as well as other components not shown in FIG. 6, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device shown in FIG. 6 is for exemplary purposes only and is not intended to limit the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 600 may also be a mobile or stationary server.

The processor 620 is configured to execute computer-executable instructions that, when executed by the processor 620, implement the steps of the data processing method described above as applied to the client management module, the initial client service module, or the update client service module.

The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the data processing method.

An embodiment of the present disclosure also provides a computer readable storage medium storing computer executable instructions that when executed by a processor implement the steps of the data processing method described above as applied to a client management module, an initial client service module, or an update client service module.

The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the data processing method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the data processing method.

An embodiment of the present disclosure further provides a computer program, where the computer program, when executed in a computer, causes the computer to perform the steps of the data processing method applied to the client management module, the initial client service module, or the update client service module.

The above is an exemplary version of a computer program of the present embodiment. It should be noted that, the technical solution of the computer program and the technical solution of the data processing method belong to the same conception, and details of the technical solution of the computer program, which are not described in detail, can be referred to the description of the technical solution of the data processing method.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the embodiments are not limited by the order of actions described, as some steps may be performed in other order or simultaneously according to the embodiments of the present disclosure. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the embodiments described in the specification.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are merely used to help clarify the present specification. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of the embodiments. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This specification is to be limited only by the claims and the full scope and equivalents thereof.

Claims

1. A data processing method applied to a data processing system, the system comprising a client management module, an initial client service module, and an update client service module, the method comprising:

the updating client service module acquires unit information of a target data processing unit from the initial client service module and calls the target data processing unit to process the data to be processed based on the unit information under the condition that the target data processing unit for processing the data to be processed does not exist;

the determining that there is no target data processing unit that processes the data to be processed includes:

2. The data processing method according to claim 1, wherein the client management module, in response to an update request, obtains the update client service module, comprising:

3. The data processing method according to claim 2, the sending the received data processing request to the update client service module, comprising:

4. The data processing method according to claim 2, further comprising, before the sending the received data processing request to the update client service module:

5. The data processing method according to claim 1, the updating client service module, before determining that there is no target data processing unit that processes the data to be processed, further comprising, before acquiring unit information of the target data processing unit from the initial client service module:

6. The data processing method according to claim 5, wherein the updating the client service module, by calling the information synchronization function provided by the initial client service module, obtains the unit information of the data processing unit from the initial client service module, includes:

7. The data processing method according to claim 6, the synchronizing unit information of a data processing unit to the update client service module, comprising:

8. The data processing method of claim 1, the obtaining unit information of the target data processing unit from the initial client service module, comprising:

and the updating client service module is used for acquiring unit information from the initial client service module by calling an information acquisition function provided by the initial client service module.

9. The data processing method according to claim 8, wherein the updating the client service module obtains the unit information from the initial client service module by calling an information obtaining function provided by the initial client service module, comprising:

the initial client service module determines unit information corresponding to the attribute information from the unit information held by the initial client service module and sends the unit information to the updating client service module;

And the updating client service module receives the unit information sent by the initial client service module.

10. The data processing method of claim 9, the sending the unit information to the update client service module, comprising:

the initial client service module performs serialization processing on the unit information held by the initial client service module to obtain a binary stream corresponding to the unit information, and sends the binary stream corresponding to the unit information to the updating client service module;

correspondingly, the updating the client service module, receiving the unit information sent by the initial client service module, includes:

And the updating client service module performs deserialization processing on the received binary stream corresponding to the unit information to obtain the unit information.

11. The data processing method of claim 1, further comprising:

The updating client service module determines the unit information and the current transmission state of the unit information, and calls the initial client service module to provide an information synchronization completion function and sends an information synchronization completion instruction to the initial client service module under the condition that the current transmission state meets the transmission completion condition;

12. A computing device, comprising:

A memory and a processor;

the memory is configured to store computer executable instructions, the processor being configured to execute the computer executable instructions, which when executed by the processor implement the steps of the data processing method of any one of claims 1 to 11, applied to a client management module, an initial client service module, or an update client service module.

13. A computer readable storage medium storing computer executable instructions which when executed by a processor implement the steps of the data processing method of any one of claims 1 to 11 for application to a client management module, an initial client service module or an update client service module.