CN112596960B - Distributed storage service switching method and device - Google Patents

Abstract

The present application relates to the technical field of data storage, and in particular to a distributed storage service switching method and device. The method includes: receiving a first data read and write request sent by a client, and determining a target storage server for processing the first data read and write request; sending the first data read and write request to a target intelligence corresponding to the target storage server A network card, so that the target smart network card performs data processing on the first data read and write request based on a distributed storage service running locally; if a failure of the target smart network card is detected, start the The distributed storage service that is set to be started, so that the target storage server performs the second data read and write request sent by the client and needs to be processed by the target storage server based on the locally running distributed storage service data processing.

Description

A distributed storage service switching method and device

technical field

The present application relates to the technical field of data storage, and in particular to a distributed storage service switching method and device.

Background technique

Distributed storage is to disperse and store data on multiple independent devices. Traditional network storage systems use centralized storage servers to store all data. Storage servers become the bottleneck of system performance and the focus of reliability and security, which cannot meet the needs of large-scale storage applications. The distributed network storage system adopts a scalable system structure, uses multiple storage servers to share the storage load, and uses the location server to locate and store information. It not only improves the reliability, availability and access efficiency of the system, but also is easy to expand.

The traditional ServerSan (software-defined storage) distributed storage architecture is: a common network card is configured on the server, and the distributed storage software runs on the x86 CPU of the host. Communication between nodes adopts TCP protocol, through the kernel network protocol stack. Distributed storage on each node manages NVMe devices local to the physical host (PCIe). Cluster linkage provides external storage services and ensures the consistency of stored data.

However, in the traditional distributed storage architecture, the distributed storage software running on the physical machine will occupy the physical machine’s cpu, memory and other system resources; sharing the physical machine’s cpu resources with the virtual machine, the number of cpus occupied by the distributed storage software will affect the The number of virtual machines that can be created. The performance of distributed storage running on a physical machine will be affected by the running of the physical machine OS and virtual machine. The distributed storage software running on the physical machine manages its local storage resources, and also needs to consume some system resources, that is, access the abstract block layer through system calls.

In order to solve the above problems, you can configure nvme disks, smart network cards, and Nvmeof targets on the storage server. There is a CPU on the smart network card that is independent of the physical server, such as an Arm CPU. Distributed storage software runs on the Arm CPU. The data disk managed by the distributed storage software is connected to the Nvmeof target of the host through Nvmeof to form a network data disk. Multiple Arm storage nodes form distributed storage.

However, the reliability of the distributed storage cluster based on the smart network card is limited by the number of Arm nodes of the smart network card, for example, the minimum number of distributed storage cluster nodes is 3. When three iNICs form a cluster environment and one iNIC fails, and there are only two nodes in the cluster, there is a risk of split-brain; when three iNICs form a cluster environment and more than two iNICs fail, the cluster cannot provide Serve.

For distributed storage with a certain number of nodes, when a small number of nodes in the cluster are down, the number of remaining storage cluster nodes can still meet the requirements. The normal processing method is to trigger data balance. When only the smart network card fails, but the disk is normal, the node cannot read and write the disk normally because of the smart network card failure. Although the cluster can still provide storage services, the new storage IO will be distributed to the disks of other nodes. Eventually, the data of the entire cluster will be unbalanced.

Contents of the invention

The present application provides a distributed storage service switching method and device, which are used to solve the problem in the prior art that the distributed storage service is unavailable due to a smart network card failure.

In the first aspect, the present application provides a method for switching distributed storage services, which is applied to a distributed storage system. Each storage server in the distributed storage system is equipped with a corresponding smart network card, and each smart network card runs a distributed storage service. service, each of the smart network cards establishes an RDMA channel with a controller for managing local storage resources on its corresponding storage server, and each of the storage servers is deployed with a distributed storage service that is set to be started, and the method include:

receiving the first data read and write request sent by the client, and determining a target storage server for processing the first data read and write request;

Sending the first data read and write request to the target smart network card corresponding to the target storage server, so that the target smart network card performs data processing on the first data read and write request based on a locally running distributed storage service;

If a failure of the target smart network card is detected, the distributed storage service deployed on the target storage server and set to be started is started, so that the target storage server Data processing is performed on the second data read and write request sent by the client and to be processed by the target storage server.

Optionally, the step of the target iNIC performing data processing on the first data read and write request based on a locally running distributed storage service includes:

The target iNIC sends the first data read and write request to a controller on the target storage server for managing local storage resources, wherein the controller executes the first data through its corresponding RDMA channel Read and write requests for data processing.

Optionally, the target iNIC writes heartbeat count information to a first designated location in the memory of the target storage server based on a preset period during normal operation;

The step of detecting the failure of the target intelligent network card includes:

When it is detected that the heartbeat count maintained at the first specified location in the internal memory of the target storage server does not increase within a preset time period, it is determined that a failure of the target smart network card is detected.

Optionally, after the distributed storage service deployed on the target storage server is started, the target storage server writes heartbeat count information to a second specified location in the memory based on a preset cycle;

When it is detected that the target iNIC is back to normal, if the state of the distributed storage service deployed on the target storage server is to be started and/or the heartbeat count maintained at a second specified location in the memory of the target storage server If there is no increase within the preset time period, the target smart network card starts the distributed storage service, and performs data processing on the third data read and write request sent by the client and needs to be processed by the target storage server.

Optionally, the method also includes:

When it is detected that the target smart network card returns to normal, if the distributed storage service deployed on the target storage server is running normally, the target smart network card sends a switching instruction to the target storage server, and starts a timer to Make the target storage server set the locally running distributed storage service to the state to be started, and start the RDMA channel with the target smart network card, and send a switching completion instruction to the target smart network card; if the target When the intelligent network card receives the switching completion instruction/does not receive the switching completion instruction when the timer times out, the target intelligent network card starts the distributed storage service, and sends the required information sent by the client. The third data read and write request processed by the target storage server performs data processing.

Optionally, the method also includes:

When the distributed storage service is started on the target iNIC, the metadata stored in the storage resource of the target storage server is loaded to a third designated location in the memory of the target storage server, wherein, when the The target smart network card is faulty, and when the distributed storage service deployed on the target storage server and set to be started is started, the target storage server reads the second data based on the metadata stored in the third specified location. Write requests for data processing.

In the second aspect, the present application provides a distributed storage service switching device, which is applied to a distributed storage system. Each storage server in the distributed storage system is respectively equipped with a corresponding smart network card, and each smart network card runs a distributed storage service. service, each of the smart network cards establishes an RDMA channel with a controller for managing local storage resources on its corresponding storage server, and each of the storage servers is deployed with a distributed storage service that is set to be started, and the device include:

A receiving unit, configured to receive a first data read/write request sent by the client, and determine a target storage server for processing the first data read/write request;

A sending unit, configured to send the first data read and write request to a target smart network card corresponding to the target storage server, so that the target smart network card performs the first data read and write request based on a locally running distributed storage service data processing;

The switching unit is configured to start the distributed storage service deployed on the target storage server and set to be started when a failure of the target smart network card is detected, so that the target storage server is based on the locally running distributed storage service. The storage service performs data processing on the second data read and write request sent by the client and to be processed by the target storage server.

Optionally, the step of the target iNIC performing data processing on the first data read and write request based on a locally running distributed storage service includes:

The target iNIC sends the first data read and write request to a controller on the target storage server for managing local storage resources, wherein the controller executes the first data through its corresponding RDMA channel Read and write requests for data processing.

Optionally, the target iNIC writes heartbeat count information to a first designated location in the memory of the target storage server based on a preset period during normal operation;

When a failure of the target intelligent network card is detected, the switching unit is specifically configured to:

When it is detected that the heartbeat count maintained at the first specified location in the internal memory of the target storage server does not increase within a preset time period, it is determined that a failure of the target smart network card is detected.

Optionally, after the distributed storage service deployed on the target storage server is started, the target storage server writes heartbeat count information to a second specified location in the memory based on a preset cycle;

The switching unit is further configured to, when detecting that the target smart network card returns to normal, if the state of the distributed storage service deployed on the target storage server is to be started and/or the status of the distributed storage service in the memory of the target storage server is The heartbeat count maintained at the second designated location does not increase within the preset time length, then the target smart network card starts the distributed storage service, and reads and writes the third data sent by the client and needs to be processed by the target storage server Request for data processing.

Optionally, the switching unit is further configured to: when detecting that the target iNIC is back to normal, if the distributed storage service deployed on the target storage server is running normally, the target iNIC sends The storage server sends a switching instruction, and starts a timer, so that the target storage server sets the locally running distributed storage service to the to-be-started state, starts the RDMA channel with the target smart network card, and sends the The target intelligent network card sends a switching completion instruction; if the target intelligent network card receives the switching completion instruction/does not receive the switching completion instruction when the timer expires, the target intelligent network card starts the distributed storage service , and perform data processing on the third data read and write request sent by the client and to be processed by the target storage server.

Optionally, the device also includes:

a loading unit, when the distributed storage service is started on the target smart network card, the loading unit loads the metadata stored in the storage resource of the target storage server to a third specified location in the memory of the target storage server, Wherein, when a failure of the target smart network card is detected and the distributed storage service deployed on the target storage server and set to be started is started, the target storage server stores the metadata based on the third specified location. Perform data processing on the second data read and write request.

In a third aspect, the embodiment of the present application provides a distributed storage service switching device, which includes:

memory for storing program instructions;

The processor is configured to call the program instructions stored in the memory, and execute the steps of the method according to any one of the above-mentioned first aspects according to the obtained program instructions.

In a fourth aspect, the embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to make the computer perform the above-mentioned first A step of any one of the methods described in the aspect.

To sum up, it can be seen that the distributed storage service switching method provided by the embodiment of the present application receives the first data read and write request sent by the client, and determines the target storage server for processing the first data read and write request; The data read and write request is sent to the target smart network card corresponding to the target storage server, so that the target smart network card performs data processing on the first data read and write request based on the distributed storage service running locally; if the target smart network card is detected If the network card fails, start the distributed storage service deployed on the target storage server that is set to be started, so that the target storage server responds to the request sent by the client based on the distributed storage service running locally. The second data read and write request processed by the target storage server performs data processing.

Using the distributed storage service switching method provided by the embodiment of the present application, when the smart network card fails to provide distributed storage service services, the distributed storage service pre-deployed on the storage server can be started, avoiding the distributed storage service caused by the smart network card failure. The problem of unavailability of the distributed storage system improves the reliability of the distributed storage system and enhances the user experience.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the application or the prior art. Obviously, the accompanying drawings in the following description These are only some embodiments described in this application, and those skilled in the art can also obtain other drawings according to these drawings of the embodiments of this application.

FIG. 1 is a schematic structural diagram of a distributed storage system provided by an embodiment of the present application;

FIG. 2 is a detailed flowchart of a distributed storage service switching method provided in an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a distributed storage service switching device provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of another distributed storage service switching device provided by an embodiment of the present application.

Detailed ways

The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. As used in this application and the claims, the singular forms "a", "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.

It should be understood that although terms such as first, second, and third may be used in the embodiment of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present application, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, furthermore, the use of the word "if" could be interpreted as "at" or "when" or "in response to a determination."

The structure of the distributed storage system provided by the embodiment of the present application will be described in detail below in combination with specific application scenarios. For example, refer to FIG. 1, which is a schematic structural diagram of a distributed storage system provided by the present application, wherein the distributed storage system includes 3 storage servers (storage server 1, storage server 2 and storage server 3) , it should be noted that, in this embodiment of the application, each storage server is deployed with a distributed storage service that is set to be started. Each storage server is configured with a corresponding smart network card (smart network card 1, smart network card 2 and smart network card 3), and each storage server includes a storage resource for storing data and a controller for managing local storage resources. To run the distributed storage service, a Remote Direct Memory Access (RDMA) channel is established between the smart network card configured on each storage server and the controller for managing local storage resources on the corresponding storage server. Storage resources managed by each controller can consist of multiple disks. Specifically, the disk may be a disk conforming to a non-volatile memory host controller interface specification (Non-Volatile Memory express, NVMe), which may be referred to as an NVMe disk in this specification.

That is to say, for a storage server, when its corresponding smart network card is normal, the distributed storage service can be provided by the smart network card. Processing, when the intelligent network card fails, the distributed storage service deployed locally by the storage server can be used to process the data read and write requests sent by the client that need to be processed by itself.

It should be noted that the above system structure, number of devices, number of disks, etc. are for illustrative purposes only, and are not intended to be limiting.

The so-called smart network card may have an independent central processing unit (Central Processing Unit, CPU). The smart network card can also be called an accelerator card, which has certain network and storage acceleration capabilities, and can usually be realized by a field-programmable gate array (Field-Programmable Gate Array, FPGA). Since the smart network card itself can process data input/output based on distributed storage, the processor of the storage server can be freed from this service, and only the smart network card can process data input/output.

Exemplarily, refer to FIG. 2 , which is a detailed flowchart of a distributed storage service switching method provided by an embodiment of the present application. The method is applied to a distributed storage system, and each storage server in the distributed storage system is configured separately There are corresponding smart network cards, and distributed storage services run on each smart network card. Each smart network card establishes an RDMA channel with the controller for managing local storage resources on the corresponding storage server. A distributed storage service in a state to be started, the method includes the following steps:

Step 200: Receive a first data read/write request sent by a client, and determine a target storage server for processing the first data read/write request.

Specifically, after the distributed storage system receives the data read and write request sent by the client, the management device as the management node will assign the data read and write request to a storage server in the distributed storage system for processing, that is, the management The device will send the data read and write request to the smart network card corresponding to the storage server for processing the data read and write request. It should be noted that the above target storage server is any storage server in the distributed storage system.

Further, in the embodiment of the present application, it is necessary to run the distributed storage service on the smart network card in advance. Specifically, the smart network card is initialized based on the user configuration, so that the above-mentioned smart network card runs the distributed storage service, and the above-mentioned smart network card corresponds to it The controller on the storage server used to manage local storage resources establishes an RDMA data channel.

Specifically, in the embodiment of the present application, the smart network card performs an initialization operation based on user configuration, so that when the above smart network card runs the distributed storage service, a preferred implementation method is that the smart network card is based on the configuration command issued by the user. Run the distributed storage service locally, and form a distributed storage service cluster with other smart network cards running the distributed storage service.

That is to say, in this embodiment of the application, the distributed storage service is run on the smart network card.

Further, the smart network card performs an initialization operation based on user configuration, so that when the smart network card establishes an RDMA data channel with the controller on the corresponding storage server for managing local storage resources, a better implementation method is that the smart network card is based on The configuration command issued by the user configures the NVMe over Fabrics protocol locally, and establishes an RDMA data channel between the initiator and the controller for managing local storage resources on the corresponding storage server. The controller that manages local storage resources is configured as the target side of the NVMe over Fabrics protocol.

Specifically, based on the configuration command issued by the user, the smart network card configures the NVMe over Fabrics protocol locally, and when the initiator side establishes a remote direct data access RDMA data channel with the controller for managing local storage resources on the corresponding storage server, A better implementation method is that the smart network card configures the IP address of RDMA locally based on the configuration command issued by the user, and as the initiator end, uses the initiator tool in the NVMe over Fabrics protocol to connect to its corresponding storage server for management A controller for local storage resources; wherein, the controller for managing local storage resources on the storage server corresponding to the smart network card configures the IP address of RDMA on the local network card interface with RDMA capability based on the configuration command issued by the user, and Configure the NVMe over Fabrics protocol on the network card chip with NVMe-oF target offlad capability as the target end. That is to say, the network card chip included in the controller on each storage server is a network card chip integrated with the RDMA function and the NVMe-oF target offflad capability.

For example, initialize the smart network card and each controller corresponding to each storage server, configure the NVMe over Fabrics protocol for each smart network card and each controller, use the smart network card as the initiator end, and connect the controller as the target end, Start the distributed storage service.

Optionally, the above initialization process may specifically include the following steps:

Step 1: Each iNIC can be configured with an area for storing metadata corresponding to data stored in its corresponding storage server, so it can also be called a metadata area. During the initialization process, the metadata stored in the storage resource is loaded into the metadata area.

Another preferred implementation manner is that an area for storing the metadata corresponding to the storage data in the storage resources on the storage server is configured in the memory of the storage server, so it may also be called a metadata area. Then, during the initialization process, the metadata in the storage resource is loaded into the metadata area of the memory.

Step 2: Each controller can be configured with an Internet Protocol address (Internet Protocol, IP) for RDMA, and configured with an NVMe over Fabrics protocol as a target end.

Step 3: Each smart network card can be configured with the IP address of RDMA and the NVMe over Fabrics protocol. As the initiator, use the initiator tool in the NVMe over Fabrics protocol to connect to its corresponding controller and start the distributed storage service.

Among them, starting the distributed storage service may specifically include: each smart network card marks and records the allocated disk locally, and writes the metadata corresponding to the storage data in the allocated disk into the metadata configured by the smart network card through the RDMA protocol in the area.

Each smart network card can perform input/output processing on the stored data through the RDMA protocol and the metadata corresponding to the stored data.

In this embodiment of the application, when enabling the RDMA data channel of each storage server, a preferred implementation method is to configure IP addresses such as 1.1.1.2, 1.1.2.2, . ..., the RoCEv2 protocol can be used.

When enabling the RDMA data channel of each smart network card, a better implementation method is to configure IP addresses 1.1.1.1, 1.1.2.1, ... for the network card interface with RDMA capability on each smart network card, and the RoCEv2 protocol can be used .

That is to say, each storage server enables the RDMA data channel, that is, configures an IP address (such as IP 11) for the network card interface with RDMA capability on the storage server 1, and enables each smart network card to enable the RDMA data channel, that is, the smart network card 1 has the The network card interface with RDMA capability is configured with an IP address (IP 21), wherein the smart network card corresponding to storage server 1 is smart network card 1, then smart network card 1 can be used as the initiator terminal, and the network card interface with the IP address configured as IP 11 and An RDMA data channel is established between network card interfaces whose IP address is configured as IP 21 on storage server 1 .

Step 210: Send the above-mentioned first data read-write request to the target iNIC corresponding to the target storage server, so that the above-mentioned target iNIC performs data processing on the above-mentioned first data read-write request based on the locally running distributed storage service.

Specifically, the management device in the distributed storage system will send the first data read and write request to the target smart network card corresponding to the target storage server for processing the data read and write request, and the target smart network card will receive the first data When reading and writing a request, data processing is performed on the first data reading and writing request based on a distributed storage service running locally.

For example, when the distributed storage service running on the target smart network card is started, if it is determined that the running state of the distributed storage service deployed on the target storage server is the waiting state, the metadata can be loaded from the disk to the target through RDMA writing. In the memory of the storage server, after the startup is completed, a distributed storage service is provided.

In the embodiment of the present application, when the target smart network card processes the above-mentioned first data read and write request based on the distributed storage service running locally, a preferred implementation method is that the above-mentioned target smart network card reads and writes the above-mentioned first data The request is sent to a controller on the target storage server for managing local storage resources, wherein the controller performs data processing on the first data read and write request through its corresponding RDMA channel.

For example, after receiving the first data read-write request, the target smart network card sends the first data read-write request to the controller in the corresponding target storage server, and when the controller receives the first data read-write request, Parse the first data read and write request, initiate a DMA operation through the RDMA data channel established with the target smart network card, and return the data read and write result to the target smart network card. That is, the target iNIC directly accesses the storage resources on the target storage server, and performs read and write operations on the storage resources.

Step 220: If the above-mentioned target smart network card failure is detected, start the distributed storage service deployed on the above-mentioned target storage server and set to be started, so that the above-mentioned target storage server can provide the above-mentioned client with the distributed storage service based on the locally running Data processing is performed on the second data read and write request sent by the terminal and required to be processed by the target storage server.

In the embodiment of the present application, a first designated location is preset in the memory of the target storage server, and when the target smart network card is running normally, it will write to the first designated location in the memory of the above-mentioned target storage server based on a preset period Heartbeat counting information; of course, a second specified location is also preset in the memory of the target storage server. After the distributed storage service deployed on the target storage server is started, the above target storage server will Periodically writes heartbeat count information to the second specified location in memory.

Further, the running state (to be started/running) of the distributed storage server deployed on the target storage server may also be written into the above-mentioned second specified location. In practical applications, the above-mentioned first specified location and the second specified location can be accessed to write/read data through memory access.

Then, when detecting the failure of the above-mentioned target smart network card, a preferred implementation method is to determine that the detection to the above target iNIC failure.

That is to say, when the distributed storage service on the target smart network card is running normally, the heartbeat count is regularly written to the first specified location in the memory of the target storage server, that is, the heartbeat count at the first specified location will change over time. increase, then, when it is detected that the heartbeat count does not increase within the preset time period, it is determined that the target intelligent network card fails.

For example, when the target intelligent network card fails (for example, software crashes), the target intelligent network card cannot write the heartbeat count to the first designated area any more. The target storage server will periodically check the heartbeat count of the first designated location, and if it is determined that the heartbeat count of the first designated location does not increase within a certain period of time, then it is determined that the target intelligent network card is abnormal. Need to enter the switching process.

Specifically, the switching process is as follows: the target storage server closes the Nvme of service; the distributed storage service that is set to be started locally is started, at this time, because the target smart network card has written the latest metadata into the In the memory, when the target storage server starts the distributed storage service, it no longer needs to load metadata from the disk; if it is determined that the startup is successful, it will regularly count the heartbeat and write it to the second designated area (which can be written through memory access), and determine The conversion is successful; further, the target storage server can also write the operating status of the locally deployed distributed storage service into the second specified area. If it is determined that the startup is unsuccessful, the heartbeat count will not be written into the second designated area periodically, and it is determined that the conversion fails.

It should be noted that, in the embodiment of the present application, when the distributed storage service is started on the above-mentioned target smart network card, the metadata stored in the storage resource of the above-mentioned target storage server is loaded to the third specified location in the internal memory of the above-mentioned target storage server , wherein, when a failure of the above-mentioned target smart network card is detected and the distributed storage service deployed on the above-mentioned target storage server and set to be started is started, the above-mentioned target storage server performs the above-mentioned metadata storage based on the metadata stored in the third specified location Two data read and write requests for data processing.

That is to say, when the smart network card is started to start the distributed storage service, the metadata in the disk is loaded to the specified location of the storage server memory, and the smart network card reads and writes the data based on the metadata stored in the specified location of the storage server memory While processing, the metadata content will be updated; when a smart network card failure is detected and the distributed storage service is switched to the storage server, the metadata stored in the specified location of the storage server memory is the latest metadata, then the storage server It is no longer necessary to perform the metadata loading operation, and directly use the metadata stored in the specified location of the storage server memory to perform data processing on subsequent data read and write requests received.

Further, when it is detected that the above-mentioned target smart network card returns to normal, if the state of the distributed storage service deployed on the above-mentioned target storage server is to be started and/or the heartbeat count maintained at the second specified location in the memory of the above-mentioned target storage server If there is no increase within the preset time period, the above-mentioned target smart network card starts the distributed storage service, and performs data processing on the third data read and write request sent by the above-mentioned client and needs to be processed by the above-mentioned target storage server.

For example, after the failure of the target intelligent network card is repaired, it is judged whether the distributed storage service deployed on the target storage server is running normally. Specifically, the running status of the distributed storage service deployed on the target In the startup state, the target smart network card directly starts the local distributed storage service. If it is in the running (started) state, it continues to obtain the heartbeat count written by the target storage server from the second designated area, and judges the target storage server based on the heartbeat count. Whether the deployed distributed storage service is running normally, if not, the target smart network card directly starts the local distributed storage service, and uses the local distributed storage service to process the subsequent data read and write requests that need to be processed by the target storage server.

Furthermore, when it is detected that the above-mentioned target smart network card returns to normal, if the distributed storage service deployed on the above-mentioned target storage server is running normally, the above-mentioned target smart network card sends a switching instruction to the above-mentioned target storage server, and starts a timer to Make the above target storage server set the locally running distributed storage service to the state to be started, and start the RDMA channel with the above target smart network card, and send the switching completion instruction to the above target smart network card; if the above target smart network card is receiving When the above-mentioned switching completion instruction is reached/the above-mentioned switching completion instruction is not received when the above-mentioned timer expires, the above-mentioned target smart network card starts the distributed storage service, and reads the third data sent by the above-mentioned client and needs to be processed by the above-mentioned target storage server. Write requests for data processing.

For example, after the failure of the target smart network card is repaired, and it is determined that the distributed storage service deployed on the target storage server is running normally, it sends the acquire command to the target storage server, starts the acquire command processing timer, and waits for the acquire finish command; the target storage server is receiving When using the acquire command, close the locally running distributed storage service, enable the NVMe-of target service, write the operating status of the locally deployed distributed storage service to the second specified area, and send the acquire finish command to the target smart network card; if If the target iNIC receives the acquire finish command before the timer expires or does not receive the acquire finish command after the timer expires, the target iNIC directly starts the local distributed storage service, and uses the local distributed storage service to meet subsequent needs. The target storage server processes the data read and write requests for data processing.

Based on the same inventive concept as the above-mentioned method embodiment, for example, refer to FIG. 3 , which is a schematic structural diagram of a distributed storage service switching device provided by the embodiment of the present application, which is applied to a distributed storage system. The distributed In the storage system, each storage server is configured with a corresponding smart network card, and distributed storage services are run on each smart network card, and each smart network card establishes an RDMA channel with a controller for managing local storage resources on the corresponding storage server, A distributed storage service that is set to be started is deployed on each storage server, and the device includes:

The receiving unit 30 is configured to receive a first data read and write request sent by the client, and determine a target storage server for processing the first data read and write request;

The sending unit 31 is configured to send the first data read and write request to the target smart network card corresponding to the target storage server, so that the target smart network card makes the first data read and write request based on the locally running distributed storage service data processing;

The switching unit 32 is configured to start the distributed storage service deployed on the target storage server and set to be started when the failure of the target smart network card is detected, so that the target storage server is based on the distributed storage service running locally. The type storage service performs data processing on the second data read and write request sent by the client and needs to be processed by the target storage server.

Optionally, the step of the target iNIC performing data processing on the first data read and write request based on a locally running distributed storage service includes:

The target iNIC sends the first data read and write request to a controller on the target storage server for managing local storage resources, wherein the controller executes the first data through its corresponding RDMA channel Read and write requests for data processing.

Optionally, the target iNIC writes heartbeat count information to a first designated location in the memory of the target storage server based on a preset period during normal operation;

When a failure of the target smart network card is detected, the switching unit 32 is specifically configured to:

When it is detected that the heartbeat count maintained at the first specified location in the internal memory of the target storage server does not increase within a preset time period, it is determined that a failure of the target smart network card is detected.

Optionally, after the distributed storage service deployed on the target storage server is started, the target storage server writes heartbeat count information to a second specified location in the memory based on a preset cycle;

The switching unit 32 is further configured to, when detecting that the target smart network card returns to normal, if the state of the distributed storage service deployed on the target storage server is to be started and/or in the memory of the target storage server If the heartbeat count maintained at the second specified location does not increase within the preset time period, the target smart network card starts the distributed storage service, and reads the third data sent by the client and needs to be processed by the target storage server. Write requests for data processing.

Optionally, the switching unit 32 is further configured to: when detecting that the target iNIC is back to normal, if the distributed storage service deployed on the target storage server is running normally, the target iNIC sends the The target storage server sends a switching instruction, and starts a timer, so that the target storage server sets the locally running distributed storage service to the to-be-started state, starts the RDMA channel with the target smart network card, and sends The target smart network card sends a switching completion command; if the target smart network card receives the switching completion command/does not receive the switching completion command when the timer times out, the target smart network card starts distributed storage service, and perform data processing on the third data read and write request sent by the client and to be processed by the target storage server.

Optionally, the device also includes:

a loading unit, when the distributed storage service is started on the target smart network card, the loading unit loads the metadata stored in the storage resource of the target storage server to a third specified location in the memory of the target storage server, Wherein, when a failure of the target smart network card is detected and the distributed storage service deployed on the target storage server and set to be started is started, the target storage server stores the metadata based on the third specified location. Perform data processing on the second data read and write request.

The above units may be one or more integrated circuits configured to implement the above method, for example: one or more specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), or, one or more microprocessors (digital signal processor, DSP for short), or, one or more Field Programmable Gate Arrays (Field Programmable Gate Array, FPGA for short), etc. For another example, when one of the above units is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU for short) or other processors that can call program codes. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC for short).

Further, for the distributed storage service switching device provided in the embodiment of the present application, from the hardware level, the hardware architecture diagram of the distributed storage service switching device can be referred to in Figure 4, and the distributed storage service switching device can Including: memory 40 and processor 41,

The memory 40 is used to store program instructions; the processor 41 invokes the program instructions stored in the memory 40, and executes the above method embodiments according to the obtained program instructions. The specific implementation manner and technical effect are similar, and will not be repeated here.

Optionally, the present application further provides a distributed storage service switching device, including at least one processing element (or chip) configured to execute the foregoing method embodiments.

Optionally, the present application further provides a program product, such as a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause the computer to execute the foregoing method embodiments.

Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium can be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage driver (such as hard disk drive), solid-state hard disk, any type of storage disk ( such as CD, DVD, etc.), or similar storage media, or a combination thereof.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more pieces of software and/or hardware.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

Moreover, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable equipment to produce computer-implemented processing, so that the information executed on the computer or other programmable equipment The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

The above is only a preferred embodiment of the application, and is not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application should be included in the application. within the scope of protection.

Claims (12)

1. A distributed storage service switching method, characterized in that it is applied to a distributed storage system, each storage server in the distributed storage system is respectively configured with a corresponding smart network card, and each smart network card runs a distributed storage service, Each smart network card establishes an RDMA channel with a controller for managing local storage resources on its corresponding storage server, and each storage server is deployed with a distributed storage service that is set to be started, and the method includes: receiving the first data read and write request sent by the client, and determining a target storage server for processing the first data read and write request; Sending the first data read and write request to the target smart network card corresponding to the target storage server, so that the target smart network card performs data processing on the first data read and write request based on a locally running distributed storage service; If a failure of the target smart network card is detected, the distributed storage service deployed on the target storage server and set to be started is started, so that the target storage server Data processing is performed on the second data read and write request sent by the client and to be processed by the target storage server. 2. The method according to claim 1, wherein the step of performing data processing on the first data read and write request by the target smart network card based on a locally running distributed storage service comprises: The target iNIC sends the first data read and write request to a controller on the target storage server for managing local storage resources, wherein the controller executes the first data through its corresponding RDMA channel Read and write requests for data processing. 3. The method according to claim 1 or 2, wherein when the target smart network card is running normally, the heartbeat count is written to a first specified location in the memory of the target storage server based on a preset cycle information; The step of detecting the failure of the target intelligent network card includes: When it is detected that the heartbeat count maintained at the first specified location in the internal memory of the target storage server does not increase within a preset time period, it is determined that a failure of the target smart network card is detected. 4. The method according to claim 3, wherein after the distributed storage service deployed on the target storage server is started, the target storage server loads The heartbeat counting information is written in the second designated location of the ; When it is detected that the target iNIC is back to normal, if the state of the distributed storage service deployed on the target storage server is to be started and/or the heartbeat count maintained at a second specified location in the memory of the target storage server If there is no increase within the preset time period, the target smart network card starts the distributed storage service, and performs data processing on the third data read and write request sent by the client and needs to be processed by the target storage server. 5. The method of claim 4, further comprising: When it is detected that the target smart network card returns to normal, if the distributed storage service deployed on the target storage server is running normally, the target smart network card sends a switching instruction to the target storage server, and starts a timer to Make the target storage server set the locally running distributed storage service to the state to be started, and start the RDMA channel with the target smart network card, and send a switching completion instruction to the target smart network card; if the target When the intelligent network card receives the switching completion instruction/does not receive the switching completion instruction when the timer times out, the target intelligent network card starts the distributed storage service, and sends the required information sent by the client. The third data read and write request processed by the target storage server performs data processing. 6. The method of claim 1, further comprising: When the distributed storage service is started on the target iNIC, the metadata stored in the storage resource of the target storage server is loaded to a third designated location in the memory of the target storage server, wherein, when the The target smart network card is faulty, and when the distributed storage service deployed on the target storage server and set to be started is started, the target storage server reads the second data based on the metadata stored in the third specified location. Write requests for data processing. 7. A distributed storage service switching device, which is characterized in that it is applied to a distributed storage system, and each storage server in the distributed storage system is respectively equipped with a corresponding smart network card, and each smart network card runs a distributed storage service, Each smart network card establishes an RDMA channel with a controller for managing local storage resources on its corresponding storage server, and each storage server is deployed with a distributed storage service that is set to be started, and the device includes: A receiving unit, configured to receive a first data read/write request sent by the client, and determine a target storage server for processing the first data read/write request; A sending unit, configured to send the first data read and write request to a target smart network card corresponding to the target storage server, so that the target smart network card performs the first data read and write request based on a locally running distributed storage service data processing; The switching unit is configured to start the distributed storage service deployed on the target storage server and set to be started when a failure of the target smart network card is detected, so that the target storage server is based on the locally running distributed storage service. The storage service performs data processing on the second data read and write request sent by the client and to be processed by the target storage server. 8. The device according to claim 7, wherein the step of performing data processing on the first data read and write request by the target smart network card based on a locally running distributed storage service comprises: The target iNIC sends the first data read and write request to a controller on the target storage server for managing local storage resources, wherein the controller executes the first data through its corresponding RDMA channel Read and write requests for data processing. 9. The device according to claim 7 or 8, wherein when the target smart network card is running normally, the heartbeat count is written to the first specified location in the memory of the target storage server based on a preset period information; When a failure of the target intelligent network card is detected, the switching unit is specifically configured to: When it is detected that the heartbeat count maintained at the first specified location in the internal memory of the target storage server does not increase within a preset time period, it is determined that a failure of the target smart network card is detected. 10. The device according to claim 9, wherein after the distributed storage service deployed on the target storage server is started, the target storage server loads The heartbeat counting information is written in the second designated location of the ; The switching unit is further configured to, when detecting that the target smart network card returns to normal, if the state of the distributed storage service deployed on the target storage server is to be started and/or the status of the distributed storage service in the memory of the target storage server is The heartbeat count maintained at the second designated location does not increase within the preset time length, then the target smart network card starts the distributed storage service, and reads and writes the third data sent by the client and needs to be processed by the target storage server Request for data processing. 11. The device according to claim 10, wherein the switching unit is further used for: When it is detected that the target smart network card returns to normal, if the distributed storage service deployed on the target storage server is running normally, the target smart network card sends a switching instruction to the target storage server, and starts a timer to Make the target storage server set the locally running distributed storage service to the state to be started, and start the RDMA channel with the target smart network card, and send a switching completion instruction to the target smart network card; if the target When the intelligent network card receives the switching completion instruction/does not receive the switching completion instruction when the timer times out, the target intelligent network card starts the distributed storage service, and sends the required information sent by the client. The third data read and write request processed by the target storage server performs data processing. 12. The device of claim 7, further comprising: a loading unit, when the distributed storage service is started on the target smart network card, the loading unit loads the metadata stored in the storage resource of the target storage server to a third specified location in the memory of the target storage server, Wherein, when a failure of the target smart network card is detected and the distributed storage service deployed on the target storage server and set to be started is started, the target storage server stores the metadata based on the third specified location. Perform data processing on the second data read and write request.

Images