CN118312337A - A remote memory access method for multi-tenant data centers - Google Patents

Abstract

The invention belongs to a data center network, and discloses a remote memory access method suitable for a multi-tenant data center, which comprises the steps that a client computer acquires necessary information of a server computer for accessing a relevant remote memory area by executing out-of-band remote procedure call; the client computer generates a client command and sends the client command to the server computer; the server side computer performs data access on the target memory based on the client side command, and feeds back the command completion condition to the client side computer after the data access; and the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition. The invention can ensure that the high priority application is not affected by the low priority application, can provide predictable delay even under high load and fault conditions, can almost immediately converge to fair bandwidth sharing under the condition of competing applications, and shortens the unavailable time during encryption key rotation.

Description

A remote memory access method for multi-tenant data centers

Technical Field

The present invention relates to a data center network, and more particularly to a remote memory access method suitable for a multi-tenant data center.

Background technique

The scale, diversity, and performance requirements of modern data center applications such as search and machine learning require networks that support high bandwidth and operation rates while achieving low tail latency. Remote direct memory access (RDMA) is an attractive option for such distributed systems because of the latency and operation rate advantages offered by single-sided reads and writes. Since these operations do not involve the remote CPU, the performance provided is only limited by the hardware.

Industry standard RDMA evolved from supercomputer environments and has been challenging to deploy in commercial data centers. RDMA assumes low-latency, reliable, ordered networks, and supercomputing fabrics implement these expectations through lossless link-level flow control enforced by switches, which allows RDMA-enabled network cards (NICs) to implement simple congestion control and loss recovery schemes to react to congestion and packet loss. These fabrics are typically single-tenant, and RDMA solutions for authorization, access control, failure recovery, and privacy protection also reflect single-tenant expectations.

In contrast, modern hyperscale data centers are characterized by multi-tenancy, where uncoordinated, massively distributed applications share common infrastructure. The diverse, time-varying application mix results in rapidly changing network traffic patterns, which require strong privacy and authentication.

But these requirements are at odds with the design choices of standard RDMA. Standard RDMA provides hardware connectivity, a capability that works well with early RDMA applications, but it has fundamental limitations on scale, isolation, performance, and fault tolerance. As modern service and storage systems scale beyond 10,000 servers, the hardware resources of each connection can be easily exhausted. Congestion control algorithms need to be constantly iterated based on deployment and application conditions, and standard direct memory access RNICs embed important parts of congestion response into hardware, which leaves little opportunity for tuning after deployment.

Since applications and infrastructure do not trust each other, multi-tenancy requires line-rate encryption as well as application support to manage the origin of encryption keys. Although modern RNICs provide encryption capabilities, practical challenges remain: encryption is tied to the connection, applications must trust lower levels of the stack to manage keys, and there is no support for security-related management operations such as encryption key rotation.

Summary of the invention

In order to solve the above technical problems, the present invention provides a remote memory access method suitable for a multi-tenant data center, which provides the performance advantages of standard one-sided RDMA, namely high bandwidth, high operation rate and low latency, while also providing predictable tail performance, scalability, fault tolerance, isolation, security features and the convenience of rapid iteration after deployment, so as to better meet the limitations of our integrated multi-tenant data center.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

The present invention is a remote memory access method applicable to a multi-tenant data center, comprising the following steps:

Step 1: The client computer obtains the necessary information for the server computer to access the relevant remote memory area by executing an out-of-band remote procedure call;

Step 2: After the client computer obtains the information required for remote access, it generates a client command and writes it into the client computer network card, and sends the client command to the server computer;

Step 3: The server computer accesses data in the target memory based on the client command, and feeds back the command completion status to the client computer after the data access;

Step 4: The client computer determines whether fault recovery and congestion control are required based on the completion status of the received command.

A further improvement of the present invention is that in step 1, the necessary information of the relevant remote memory area obtained by the client computer by executing an out-of-band remote procedure call includes: an encryption key K_d and a structure name RegionId of the memory to be accessed.

A further improvement of the present invention is that: in the step 2, the network card includes a registration area table, a command slot and a command slot table, a request window and a first-in-first-out arbiter. The registration area table is a memory conversion table for remote memory access. The architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the computer memory range where the network card is located and all corresponding metadata, namely, the region key K_r, PCIe address, boundary, and permission. Each time the server-side computer memory is accessed, it represents a memory area in the registration area table. The command slot represents an operation being executed and can be reused after the operation is completed. The command slot table is composed of a fixed number of slots in the network card, and each slot is uniquely identified by its command slot number. The network card encodes the slot when the operation is completed. The encoding indicates to the client computer operating system which command has been completed, because the operation may be completed out of sequence. The capacity in the request window is dynamically shared by a pair of first-in-first-out arbiters, that is, each internal remote memory access service category is selected by a first-in-first-out arbiter from the ready commands in the registration area table.

A further improvement of the present invention is that in step 2, after the client computer obtains the necessary information required for remote access, the client computer generates a client command specifically as follows: the client computer writes a client command to the network card through PCIe according to the necessary information required for remote access, and writes the MMIO storage into the command slot on the network card, the command slot will be queued and wait for the service operation, thereby starting the required RMA operation, and starting the required RMA operation specifically includes: putting the command slot into service, the RMA operation waits for the capacity of the request window in the network card to meet its own needs, and sends a request to the server-side computer. After the RMA operation enters service, the network card will borrow the request window according to the actual size, and use the encryption key K_d to sign the request sent by the RMA operation to the server-side computer, and the encryption key K_d is provided in the RMA operation command and sent to the network.

A further improvement of the present invention is that in step 3, the server-side computer performs data access to the target memory based on the client command as follows: the request reaches the network card of the server-side computer, the network card consults a fixed-size on-chip table, that is, a table integrated in the chip that can query the location of the memory RegionId, searches for key information of the architecture name RegionId of the memory to be accessed contained in the request, derives the encryption key K_d, and verifies the inbound data packet. After successful verification, the network card accesses the memory through PCIe and executes the client command.

A further improvement of the present invention is that in step 3, after data access, the server-side computer feeds back the command completion status to the client computer, specifically including: the server-side computer network card returns a completion indicator to the completion area specified by the client computer that initiated the command, wherein the completion indicator includes the command slot number of the completion command slot, the time required to execute the operation, the time required to request to enter the launcher service and an operation status code, and the operation status code includes successful completion, NACK remote congestion, and TIMEOUT timeout.

A further improvement of the present invention is that in step 4, the client computer determines whether fault recovery is required based on the completion status of the received command. Specifically, the client computer generates accurate and fast feedback in the form of a clear fault code based on the time required to execute the operation, the time required to request to enter the launcher service, and the operation status code in the received completion indicator, indicating to the client computer which fault mode has occurred and the appropriate action to be taken by the client computer when encountering such a fault.

A further improvement of the present invention is that in step 4, the client computer determines whether congestion control is needed based on the completion status of the received command, including: the client computer makes a policy decision to adjust the operation release rate based on the time required to execute the operation in the received completion indicator, the time required to request to enter the launcher service, and the failure result.

A further improvement of the present invention is that remote access between a client computer and a server computer is guided by out-of-band encrypted RPC, and the RPC includes a secure exchange of keys, specifically: when registering memory, a region is assigned an architecture name RegionId of the memory to be accessed, and an application on a client computer specifies a region key K_r to protect the corresponding memory region. K_r is a 128-bit value from which an encryption key K_d is derived. The encryption key K_d constitutes the basis of the security of the RMA operation and protects a single transmission. Neither the region key K_r nor the encryption key K_d will be sent over the network as part of the remote memory access operation or its response.

A further improvement of the present invention is that an encryption key K_d is derived to generate a message authentication code, sign all protocol messages, and encrypt all data related to the transmission. The calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType).

The present invention is completely different from standard remote direct memory access (RDMA). The network card hardware in the present invention is very simple and only focuses on fast, fixed functional primitives, which is different from RDMA that provides the illusion of unlimited resources. The present invention manages explicitly limited hardware resources in the client computer. In order to facilitate fast iteration, the client computer also implements the functions of fault recovery, congestion control and inter-operation ordering.

The remote memory access method (SRMA) of the present invention is connectionless, and the state of the client computer network card will not grow as the number of endpoint pairs increases. By getting rid of connection semantics, the network card (NIC) can treat each operation as independent of other operations, allowing the client computer operating system to handle the order between operations when necessary. The remote memory access method of the present invention assigns the task of retrying and fault recovery of each operation to the client computer operating system, and provides a simpler fault fast recovery behavior. The client computer network card ensures timely completion and directly provides a fast and concise operation fault notification to the application of the client computer.

The remote memory access methods of the present invention are all small-scale, and take requests as the basis for all transmissions. Each SRMA operation transmits a maximum of 4KB of payload, thereby achieving isolation and priority sorting; in addition, all data transmissions require requests: SRMA will not start transmission unless it guarantees that the data will be accessed to the network card NIC SRAM of the present invention. The client computer network card forced request design can prevent the formation of large inconsistencies, and combining requests with small operations can achieve fast response congestion control and limit network queuing.

The client computer network card can assist the client computer operating system in congestion control, which is in contrast to the existing RDMA scheme that embeds the congestion control algorithm into the client computer network card. The present invention provides latency information for each operation and clear fault fast notification for congestion events, so that software can easily distinguish between local and remote congestion and take precise actions to minimize congestion, timeouts and packet losses.

The present invention allocates resources based on business priorities and sets clear limits on the work that the client computer operating system can provide at a time, turning limited resource pools into advantages. These limits, combined with small-scale operations, can prevent low-priority applications from monopolizing the network, and since there is no need to provide the illusion of near-infinite resources, planning operations in an explicit limited resource pool can simplify client computer network cards.

The transmission of the present invention is encrypted and signed in the client computer network card using line rate AES-GCM, and the design is consistent with the connectionless architecture of the present invention. The present invention allows the client computer's application to directly manage encryption keys without extending trust in infrastructure software, and supports frequent encryption key rotation while minimizing availability interruptions.

The beneficial effects of the present invention are as follows: the remote memory access method of the present invention divides large operations into 4KB operations, and implements congestion control and operation management in the operating system of the client computer, so that only 0.5 cores are needed to drive a line speed of 100Gbps, so that the memory access method of the present invention can provide predictable delays even under high load and fault conditions, ensuring that high-priority applications are not affected by low-priority applications, and congestion control can almost instantly converge to fair bandwidth sharing in the presence of competing applications, and shorten the unavailable time during encryption key rotation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a remote memory access method according to the present invention.

FIG. 2 is a schematic diagram of the client computer operating system of the present invention.

FIG3 is a schematic diagram of the main components of the network card of the remote memory access method of the present invention.

FIG. 4 is a schematic diagram of the overall operation of the remote memory access method of the present invention for executing a 2KB read operation.

FIG5 is a schematic diagram of the generation and distribution of security keys of the remote memory access method of the present invention.

Detailed ways

The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

The present invention proposes a remote memory access method suitable for a multi-tenant data center, which divides responsibilities for hardware and software.

As shown in Figure 2, the client computer operating system of the remote memory access method of the present invention provides a large amount of transmission abstraction and congestion management. At the bottom layer, the command entry object manages a set of command slots and a memory area configured to accept completions, providing a familiar command/completion queue structure. Since the command slots are memory-mapped registers in the network card NIC, the command entry handles the details of mmap() so that these registers are inserted into the application memory of the client computer. Based on the command entry, the next layer of the software stack, the command executor, supports transmissions of any size, which are transparently divided into operations of up to 4KB in size and are affected by congestion control software adjustments. The application software layer of client computing above the command executor is not responsible for block division, speed regulation or congestion control. The application can choose to manage faults according to personal needs. Since the network card of the present invention does not retain the state of each destination, each command fully encodes all metadata required for the operation.

As shown in FIG3 , the network card of the present invention includes a registration region table, a command slot and a command slot table, a request window, and a first-in-first-out arbiter. The registration region table is a memory conversion table for remote memory access. The architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the computer memory range where the network card is located and all corresponding metadata, namely, the region key K_r, PCIe address, boundary, and permission. Each time the server-side computer memory is accessed, it represents a memory region in the registration region table. The command slot represents an operation being executed and can be reused after the operation is completed. The command slot table is composed of a fixed number of slots in the network card, and each slot is uniquely identified by its command slot number. The network card encodes the slot when the operation is completed. The encoding indicates to the client computer operating system which command has been completed because the operation may be completed out of sequence. The capacity in the request window is dynamically shared by a pair of first-in-first-out arbiters, namely, each internal remote memory access service category is selected by a first-in-first-out arbiter from the ready commands in the registration region table.

When the application issues an operation to a command slot, the CPU performs an MMIO write operation to the corresponding hardware register through PCIe. This write causes the SRMA hardware pipeline to queue in the hardware first-in-first-out according to the service class of the operation, waiting to be scheduled. The head of the queue waits for capacity in the request window and then sends the request to the remote end, which is specified by the address information in the command itself.

Taking a 2KB read operation as an example, as shown in FIG1 , the remote memory access method applicable to a multi-tenant data center of the present invention specifically includes the following steps:

Step 1: The client computer obtains the necessary information for the server computer to access the relevant remote memory area by executing an out-of-band remote procedure call.

Before performing the RMA operation, the client computer is started to perform an out-of-band remote procedure call (out-of-band RPC) to obtain the necessary information required for the remote memory area to be accessed, including: the encryption key K_d, which is the encryption security key bound to the startup client, which is difficult to guess and cannot be transferred, and RegionId, which is the architecture name of the memory to be accessed established when registering the memory on the server. All protocol messages are signed with the information verification code generated by the encryption key K_d. Similarly, all data is encrypted using the encryption key K_d.

Step 2: After the client computer obtains the information required for remote access, it generates a client command, writes it into the client computer network card, and sends the client command to the server computer.

After obtaining the information required for remote access, the client initiates its required RMA operation, which is a 2KN read operation. In this case, a 2KB read operation is performed by writing a client command on PCIe and storing MMIO into a command slot on the network card NIC, which will be queued for service operations. The required RMA operation is thus initiated, and according to the bidding rules of SRMA, the request is waiting for execution. To enter the service, the operation needs to obtain 4KB of free space in the on-chip request window, which is an SRAM buffer used to store inbound payloads. Regardless of the size of the operation, 4KB needs to be arbitrated, which prevents small operations from occupying the space of large operations, making large operations hungry.

After the RMA operation enters service, the network card will deduct the actual size, i.e., the 2KB borrowing request window, and use the encryption key K_d to sign the RMA operation request sent to the server computer. The encryption key K_d is provided in the RMA operation command and sent to the network.

Step 3: The server computer accesses data to the target memory based on the client command, and feeds back the command completion status to the client computer after the data access. The server computer network card returns a completion indicator to the designated completion area of the client computer that initiated the command, wherein the completion indicator includes the command slot number of the completion command slot, the time required to perform the operation, the time required to request to enter the initiator service, and an operation status code, wherein the operation status code includes successful completion, NACK remote congestion, and TIMEOUT timeout.

Typically, a request arrives at the server-side computer's network card, which consults a fixed-size on-chip table, a table integrated into the chip that can query the memory location of the RegionId, to find the RegionId contained in the request, derive the encryption key K_d, and verify the inbound data packet. After successful verification, the network card accesses the memory through PCIe and executes the client command. The NIC then reads the requested data through PCIe and uses the data stream after the read as a separate network response stream. Each response is encrypted and signed with the previously derived key information. Due to adaptive routing, these responses travel through the network, but may arrive out of order.

Upon arrival at the initiator, each response is individually verified and decrypted, then transferred to the initiating host's memory via the PCIe write stream, with the offset of each response encoded in the inbound response. To avoid out-of-order responses, the NIC tracks byte arrivals and, in the error-free case, writes a successful operation completion to the initiator software once all bytes have arrived. The operation completion also includes hardware latency metrics showing how long it took to perform the operation (total_delay) and how long it took the request to get serviced by the initiator (issue_delay).

Although a failure-free scenario has been described, the above operations may experience various failures. For example, a read operation may not be serviced at the initiator due to severe local congestion, such as a large number of queued operations. The service NIC may drop or "NACK" requests when requests arrive at an overloaded server, such as an excessively long inbound request queue. Finally, responses from the server to the client may be dropped or delayed in the network.

Step 4: The client computer determines whether fault recovery and congestion control are required based on the completion status of the received command.

The present invention generates precise rapid feedback in the form of clear fault codes, indicating to the startup software which fault mode has occurred. When such a fault is encountered, the application software can take appropriate actions, such as immediately retrying the operation.

Given that the present invention is targeted for deployment on mutually untrusted endpoints and mutually untrusted network structures, all fundamental aspects of the remote memory access method of the present invention include security.

As shown in FIG5 , the security of the present invention is as follows:

Similar to standard RDMA, the present invention uses out-of-band encrypted RPC to guide remote access between a client computer and a server computer, wherein the RPC includes a secure exchange of keys, specifically: when registering memory, a region is assigned an architectural name RegionId of the memory to be accessed, an application on a client computer specifies a region key K_r to protect the corresponding memory region, K_r is a 128-bit value from which an encryption key K_d is derived, the encryption K_d constitutes the basis of the security of RMA operations and protects a single transmission, neither the region key K_r nor the encryption key K_d will be sent over the network as part of a remote memory access operation or its response.

The encryption key K_d is derived to generate a message authentication code, sign all protocol messages, and encrypt all data related to the transmission. The calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType).

The key calculated by this function is specific to each initiating process (identified by the process's host address and the process's PID) and operation type, and the key K_r rooted in a single shared process is easily placed in each region table in the NIC memory. When the client computer's application allocates a command slot to initiate RMA, the SRMA driver always stores the associated PID_Initiator in the slot's configuration. The SRMA hardware always includes the PID_Initiator as well as the RegionId in the request packet. Therefore, the server-side computer network card can derive the assigned encryption key K_d for each inbound request. In addition, key derivation can be easily completed on the server-side computer operating system, and the server operating system can calculate K_d for any potential initiator and send K_d to the initiator in the RPC response without retaining K_d on any host or NIC table. The present invention adds a message counter that increments by NIC to the encryption process to protect key integrity and prevent replay. Crucially, the security-related state on the NIC does not grow with the number of communication endpoint pairs.

Faced with a malicious user trying to access a remote memory region owned by another tenant, a process attacker can easily guess the RegionId, but it is difficult to obtain a K_d to match the host and process, short of a root-level attack. Faced with an attacker with full access to network links and switches, they can observe the ciphertext during transmission, but it is not easy to decrypt it without compromising the participating hosts. It is worth noting that the present invention can also prevent repeated attacks. Although the remote memory access method of the present invention does not prevent repeated read requests from being accepted, such attacks will generate new encrypted ciphertexts, so the attacker still needs the correct password K_d to decrypt the resulting response.

The security model of the present invention does not alter the behavior of a host with a root-level compromise, where a root-level attacker can emulate processes on the host and therefore authenticate as one of these users in any manner of their choosing. The primary defense against this threat is to accelerate detection, for example, by performing frequent encryption key rotation, enforcing repeated authentication steps that can be logged and inspected. The present invention provides explicit support for encryption key rotation to help achieve this goal.

If verification fails, the network card of the present invention will discard the response of the verification failure. In contrast, an inbound request that fails to pass the identity authentication will immediately send a failure notification as a response and use a known reserved key signature, resulting in the generation of a REMOTE_AUTHENTICATION_FAILURE signal. Discarding such requests will make the remote memory access of the present invention more powerful, forcing the attacker to face a timeout instead of making a timely negative response. However, doing so will have an impact on non-attack situations because the remote authentication failure error code is easily identified as a side effect of encryption key rotation, so the recovery steps are obvious to the client software. In contrast, the discarded request appears as a timeout, and the timeout does not immediately indicate that an encryption key rotation may have occurred.

The present invention can ensure that high-priority applications are not affected by low-priority applications, provide predictable delays even under high load and failure conditions, and congestion control can almost instantly converge to fair bandwidth sharing in the presence of competing applications, shortening the unavailability time during encryption key rotation.

The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims (10)

1. A remote memory access method suitable for a multi-tenant data center is characterized in that: the remote memory access method specifically comprises the following steps:

step 1, a client computer acquires necessary information of a server computer for accessing a related remote memory area by executing out-of-band remote procedure call;

Step 2, after the client computer obtains the information required by remote access, generating a client command, writing the client command into a client computer card, and sending the client command to the server computer;

step 3, the server computer performs data access on the target memory based on the client command, and feeds back the command completion condition to the client computer after the data access;

And step 4, the client computer judges whether fault recovery and congestion control are needed according to the received command completion condition.

2. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step 1, the necessary information of the relevant remote memory area acquired by the client computer through executing the out-of-band remote procedure call includes: encryption key k_d, architecture name RegionId of the memory to be accessed.

3. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in step 1, a remote access between a client computer and a server computer is guided by an out-of-band remote procedure call comprising a secure exchange of keys, in particular: in memory registration, a region is assigned a frame name RegionId of the memory to be accessed, an application of a client computer designates a region key k_r, protecting the corresponding memory region, k_r is a 128-bit value from which an encryption key k_d is derived, the encryption k_d forms the basis of the security of the RMA operation, and protects a single transmission, and neither the region key k_r nor the encryption key k_d is sent over the network as part of a remote memory access operation or its response.

4. A remote memory access method for a multi-tenant data center according to claim 3, wherein: the derived encryption key k_d is used to generate a message authentication code, sign all protocol messages, and encrypt all data related to transmission, and the calculation formula is as follows:

K_d＝AES(Key＝K_r,Contents＝Address_Initiator,PID_Initiator,OperationType)。

5. the method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step2, the network card includes a registration area table, a command slot and a command slot table, a request window and a fifo arbiter, where the registration area table is a memory conversion table accessed by remote memory, the architecture name RegionId of the memory to be accessed indexes the memory conversion table to display the computer memory range where the network card is located and all corresponding metadata, i.e., area keys k_r, PCIe addresses, boundaries and permissions, each time the server computer memory is accessed, the network card represents a memory area in the registration area table, the command slot represents an operation in execution, the operation can be reused after completion, the command slot table is composed of a fixed number of slots in the network card, each slot is uniquely identified by its command slot number, the network card encodes the arbitration slot when the operation is completed, the encoding indicates to the client computer operating system which command has been completed, and the capacity in the request window is dynamically shared by a pair of fifos, i.e., each internal remote memory access service class is selected by a fifo in the ready command in the registration area table.

6. The method for remote memory access for a multi-tenant data center of claim 5, wherein: in the step 2, the client computer obtains the necessary information required by remote access and then generates a client command specifically as follows: the client computer writes a client command into the network card through PCIe according to the necessary information required by remote access, and writes MMIO storage into a command slot on the network card, wherein the command slot queues enqueues to wait for service operation, thereby starting the required RMA operation, and the starting the required RMA operation specifically comprises: enabling the command slot to enter service, enabling the RMA operation to wait for the capacity of a request window in the network card to meet the requirement of the user, sending a request to a server-side computer, borrowing the request window according to the actual size by the network card after the RMA operation enters service, signing the request sent to the server-side computer by the RMA operation by using an encryption key K_d, wherein the encryption key K_d is provided in the RMA operation command and sent to a network.

7. The method for remote memory access for a multi-tenant data center of claim 1, wherein: in the step 3, the data access of the server computer to the target memory based on the client command is specifically: the request reaches the network card of the server computer, the network card refers to the on-chip table with fixed size, the key information of the architecture name RegionId of the memory to be accessed contained in the request is searched, the encryption key K_d is deduced, the inbound data packet is verified, after the verification is successful, the network card accesses the memory through PCIe, and the client command is executed.

8. The method for remote memory access for a multi-tenant data center of claim 7, wherein: in the step 3, the feedback command completion situation from the server computer to the client computer after the data access specifically includes: the server computer card returns a completion indicator to the client computer designated completion area that initiated the command, wherein the completion indicator includes the command slot number to complete the command slot, the time required to perform the operation, the time required to request access to initiator services, and an operation status code that includes successful completion, NACK remote congestion, TIMEOUT.

9. The method for remote memory access for a multi-tenant data center of claim 8, wherein: in the step 4, the step of determining, by the client computer, whether the fault recovery is required according to the received command completion condition is specifically: the client computer generates accurate and fast feedback in the form of an explicit fault code based on the time required to perform the operation, the time required to request access to the initiator service, and the operation status code received in the completion indicator, indicating to the client computer which fault mode occurred and the appropriate action the client computer took when such a fault was encountered.

10. The method for remote memory access for a multi-tenant data center of claim 9, wherein: in the step 4, the client computer determines whether congestion control is required according to the received command completion condition, including: the client computer makes policy decisions to adjust the operation release rate based on the time required to perform the operation, the time required to request access to the initiator service, and the failure results in the received completion indicator.