CN120017600B - A message processing system - Google Patents

Abstract

The request message verification module in the message processing system provided by the invention verifies the received request message and maintains QP state information, screens out the request message passing verification, the RQE processing verification module verifies RQE and maintains QP state information, judges whether the RQE space is enough to distribute the request message, the request message load uploading module writes the request message load into a host and maintains QP state information according to the remote address or RQE address of the request message, and the completion verification module determines and informs the request message processing mode according to the QP state information so as to maintain QP state information synchronization, and the response construction module constructs a response message. The processing system can further decouple each processing operation through the QP state information maintenance and synchronization mechanism, avoid the similar head blocking problem of each module, realize the full pipelining processing of the RDMA network card on the request message and improve the processing performance of the response end of the RDMA network card.

Description

A message processing system

Technical Field

The present invention relates to the technical field of remote direct memory access, and in particular to a message processing system.

Background Art

In recent years, with the rapid development of technologies such as artificial intelligence and cloud computing, RoCEv2 (RDMA over ConvergedEthernet) technology has been widely used in data centers, and many domestic and foreign manufacturers have launched their own RDMA-related IPs and products. However, on the one hand, although there is a series of open source support for RDMA-related software, such as OFED (OpenFabrics Enterprise Distribution), major manufacturers still do not disclose their hardware implementation architecture; on the other hand, for RDMA network cards deployed in data centers, the link bandwidth they support is mostly at the level of hundreds of Gbps. For FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) that implements RDMA network cards, this order of magnitude of data processing needs require precise pipeline design to meet; Taking the processing flow of the Send request message of the RC (Reliable Connection) service type at the RDMA network card responder as an example, based on the IB (InfiniBand) protocol requirements, its processing includes at least the verification of a series of information such as Opcode and PSN (Packet Sequence Number), obtaining RQE (Receive Queue Element) from the host, accumulating message length detection, sending message data to the host, generating CQE (Completion Queue Element) and other information, and so on. Element, complete queue element) to the host and many other steps, and the message processing of the same QP may have similar head blocking problems, which greatly increases the coupling between the various steps. For example, only when the previous message of QP (Queue Pai, queue pair) completes the update of ePSN (Expected Packet Sequence Number) can the next message use this ePSN to decide its processing method, and the maximum processing delay of each message requires hundreds of clock cycles. Obviously, a simple message processing mechanism is difficult to meet the bandwidth requirements of hundreds of Gbps.

Therefore, providing a message processing system that solves the above problems is an urgent problem to be solved by those skilled in the art.

Summary of the invention

The object of the present invention is to provide a message processing system, which has a simple structure, is safe, effective, reliable and easy to operate, and realizes hardware pipeline processing to improve the processing performance of the RDMA network card responder under the premise of meeting the requirements of the IB protocol.

Based on the above objectives, the technical solution provided by the present invention is as follows:

A message processing system, comprising:

A request message verification module, used to verify the received request message and maintain the first QP state information, and filter out the request message that passes the verification;

An RQE processing and verification module, used to verify the RQE obtained from the host and maintain the second QP state information, and verify whether the RQE space is sufficient to be allocated to the request message;

A request message payload upload module, used to write the request message payload to the host and maintain the third QP state information according to the remote address in the request message or the address in the RQE;

a completion verification module, configured to determine a request message processing mode according to the first QP state information, the second QP state information, and the third QP state information, and notify the request message processing mode to maintain synchronization of the first QP state information, the second QP state information, and the third QP state information;

The response message construction module is used to construct a response message according to the request message processing method.

Preferably, the request message verification module includes: a sending verification module, a rationality verification module, a first screening module and a first QP state module;

The sending verification module is used to verify whether the received request message is the request message of the message processing system;

The rationality checking module is used to check whether the request message is reasonable;

The first screening module is used to screen the request messages whose judgment results of the sending verification module and the rationality verification module are both yes;

The first QP status module is used to mark the request message as abnormal and update the first QP status information when either the judgment result of the sending verification module or the judgment result of the rationality verification module is negative.

Preferably, the first QP state module is further configured to process the request message according to the RC type QP hardware state when the QP type in the first QP state information is the RC type;

The first QP state module includes: a first state switching submodule;

The first state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the judgment result of the rationality check module is no, switch to the wait_send_nack state;

If the completion check module notifies that the request message is a Class B error defined by the IB protocol, it switches to the wait_epsn state;

If the request message triggers a Class A or Class C error defined in the IB protocol or a host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, the state is switched to the error state.

Preferably, the first QP state module is further configured to:

When in error state, the network card is called to directly and silently discard all the request messages;

When in the normal state, the network card is called to discard the request message that is retransmitted in error, and the remaining request messages are marked according to the IB protocol and the judgment result of the rationality check module;

When in the wait_send_nack state, calling the network card to process the correctly retransmitted request message and discarding the remaining request messages;

When in the wait_epsn state, the network card is called to process the request message whose judgment result of the rationality check module is yes or correctly retransmitted, and the remaining request messages are discarded, wherein the processing method of the request message whose judgment result of the rationality check module is yes is consistent with the processing method when in the normal state.

Preferably, the RQE processing verification module includes: a cache module, an RQE state verification module, an RQE space verification module, a second screening module and a second QP state module;

The cache module is used to obtain the RQE of the active QP from the host and cache it to the network card in different QPs;

The RQE status verification module is used to verify whether the RQE is correct, if not, it is marked as a malformed RQE;

The RQE space check module is used to check whether there is an RQE corresponding to the QP cached in the network card according to the request message, and if so, allocate the RQE corresponding to the QP cached to the request message; if not, trigger an RNR error and mark the request message;

The second screening module is used to screen the request message whose judgment result of the RQE state verification module and the judgment result of the RQE space verification module are both yes;

The second QP status module is used to mark the request message as abnormal and update the second QP status information when either a judgment result of the RQE status check module or a judgment result of the RQE space check module is negative.

Preferably, the second QP state module is further configured to process the request message according to the RC type QP hardware state when the QP type in the second QP state information is the RC type;

The second QP state module includes: a second state switching submodule;

The second state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the judgment result of the RQE space check module is no or the request message has been marked as a Class B error defined by the IB protocol, it switches to the wait_send_nack state;

If the completion check module notifies that the request message is a Class B error defined by the IB protocol, it switches to the wait_epsn state;

If the request message has been marked as a Class A or Class C error defined in the IB protocol, or the host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, then the state is switched to the error state.

Preferably, the second QP state module is further used for:

When in error state, the network card is called to directly and silently discard all the request messages;

When in the wait_send_nack state, calling the network card to process the correctly retransmitted request message and discarding the remaining request messages;

When in the normal state or the wait_epsn state, if the request message triggers an RNR error and the request message has been marked as a Class B error with a negative judgment result by the rationality check module, the request message is re-marked as an RNR error.

Preferably, the RQE space check module further comprises: a double check submodule;

The double check submodule is used to check whether the RQE corresponding to the QP is deformed and whether the remaining host memory space corresponding to the RQE supports the writing of the current request message payload data after the RQE corresponding to the QP already exists in the network card;

If the RQE is marked as malformed, marking the request message as a malformed RQE error;

If the remaining host memory space corresponding to the RQE does not support the writing of the current request message payload data, the request message is marked as an invalid request error.

Preferably, the second QP state module further includes: a third state switching submodule;

The third state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the request message has been marked as a Class A or Class C error defined by the IB protocol or a host configuration QP error, or the completion check module notifies a Class A or Class C error defined by the IB protocol, or the request message is marked as either the malformed RQE error or the invalid request error, then switch to the error state.

Preferably, the request message load uploading module includes: a load uploading module, a load data writing module and a third QP state module;

The load uploading module is used to determine whether the request message needs to be loaded and uploaded;

The load data writing module is used to, when the load uploading module determines that the result is yes, obtain the corresponding physical page address from the host according to the remote address in the request message or the address in the RQE, write the request message load into the host memory according to the physical page address, and construct a request message descriptor;

The third QP state module is configured to process the request message according to the RC type QP hardware state when the QP type in the third QP state information is the RC type.

Preferably, the third QP state module comprises: a fourth state switching submodule;

The fourth state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the request message has been marked as a Class A or Class C error defined in the IB protocol, or the host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, then the state is switched to the error state.

Preferably, the completion verification module includes: a request message processing mode determination module and a notification module;

The request message processing mode determining module is used to determine the request message processing mode according to the first QP state information, the second QP state information and the third QP state information included in the request message descriptor;

The notification module is used to send the message information in the request message descriptor to the request message verification module, the RQE processing verification module and the request message load upload module respectively, so as to update the corresponding QP status information.

Preferably, the response message construction module is specifically used to generate a response message descriptor according to the request message descriptor, and construct a response message according to the response message descriptor;

If the response message is a read response message, the response message construction module is further specifically used to construct the response message according to the response message descriptor, the remote address in the request message, and the physical page address.

The message processing system provided by the present invention verifies the received request message and maintains the QP status information through the request message verification module, and selects the request message that passes the verification; verifies the RQE and maintains the QP status information through the RQE processing verification module, and determines whether the RQE space is sufficient to allocate the request message; writes the request message load to the host and maintains the QP status information according to the request message remote address or RQE address through the request message load upload module; determines and notifies the request message processing mode according to the QP status information of the above module through the completion verification module to maintain the synchronization of the QP status information, and constructs the response message through the response construction module. Through the QP status information maintenance and synchronization mechanism, each processing operation can be further decoupled to avoid the occurrence of similar head blocking problems caused by each module to realize abnormal situation processing, and the full pipeline processing of the RDMA network card for the request message can be realized, and the processing performance of the RDMA network card response end can be improved.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of the structure of a message processing system provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a request message verification module provided in an embodiment of the present invention;

FIG3 is a schematic diagram of the structure of an RQE processing and verification module provided in an embodiment of the present invention;

FIG4 is a schematic diagram of the structure of a request message load upload module provided in an embodiment of the present invention;

FIG5 is a schematic diagram of the structure of a completion verification module provided in an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The embodiments of the present invention are written in a progressive manner.

The embodiment of the present invention provides a message processing system, which mainly solves the technical problem in the prior art that each module implements abnormal situation processing, resulting in similar head blocking problems and reducing the processing performance of the RDMA network card responder.

As shown in FIG1 , a message processing system includes:

A request message verification module, used to verify the received request message and maintain the first QP state information, and filter out the request message that passes the verification;

An RQE processing and verification module is used to verify the RQE obtained from the host and maintain the second QP state information, and to verify whether the RQE space is sufficient to be allocated to the request message;

A request message payload upload module, used to write the request message payload to the host and maintain the third QP state information according to the remote address in the request message or the address in the RQE;

A completion verification module is used to determine a request message processing mode according to the first QP state information, the second QP state information, and the third QP state information, and to notify the request message processing mode to maintain synchronization of the first QP state information, the second QP state information, and the third QP state information;

The response message construction module is used to construct the response message according to the request message processing method.

During actual use, the request message verification module receives the request message transmitted by the network card, verifies the message and maintains the first QP state information, and selects the request message that passes the verification; the request message that passes the verification is sent to the RQE processing verification module, and the RQE processing verification module obtains the RQE of the active QP from the host in advance, verifies the RQE and maintains the second QP state information, and verifies whether the space of the RQE is sufficient to be allocated to the request message that passes the verification; if so, the request message payload upload module writes the request message payload to the host and maintains the third QP state information according to the remote address in the request message or the address in the RQE; the completion verification module receives the QP state information maintained by the above module, determines the request message processing method and sends it to the response message construction module, and notifies the above module, so that the QP state information of each module is synchronized; the response construction module constructs a response message according to the request message processing method.

Preferably, the request message verification module includes: a sending verification module, a rationality verification module, a first screening module and a first QP state module;

A sending verification module is used to verify whether the received request message is a request message of the message processing system;

A rationality check module is used to check whether the request message is reasonable;

The first screening module is used to screen the request messages whose judgment results of the sending verification module and the rationality verification module are both yes;

The first QP state module is used to mark the request message as abnormal and update the first QP state information when either the judgment result of the sending verification module or the judgment result of the rationality verification module is negative.

In actual use, when screening RoCEv2 messages, first determine whether it is a RoCEv2 message sent to the local end based on the Ethernet layer 2, 3, and 4 header information of the message, and then determine whether it is a reasonable RoCEv2 request message based on the BTH (Base Transport Header) header information of the IB protocol and the corresponding QPC (Queue Pair Context) information. Only reasonable RoCEv2 request messages will be processed by the above subsequent modules, otherwise they will be directly silently discarded or sent to other processing logic of the network card according to the protocol requirements;

The preliminary verification of the RoCEv2 request message refers to the verification that can be directly judged based on the BTH and ETH (Extended Transport Header) information of the message, QPC, MRC (Memory Region Context) and the QP-related status information maintained by the module itself. If the preliminary verification passes, the processing of the subsequent modules will continue, otherwise the relevant exception handling operations will be performed according to the cause of the error. The contents of the preliminary verification include: whether the Opcode type is supported, the correctness of the Opcode sequence, the correctness of the PSN, whether the R_Key (Remote Key) matches the access requirement, the correctness of the Q_Key (Queue Key), whether the message length matches the PMTU (Path Maximum Transmission Unit), the correctness of the Pad Count, etc. Among them:

Opcode type check is used to check whether the QP corresponding to the message supports the request type corresponding to the Opcode field in the BTH header information of the message;

Opcode sequence check is used to check whether the Opcode of the message is in sequence with the Opcode of the last correct message of the corresponding QP; therefore, the request message check module needs to record the Opcode of the last correct message of each QP;

PSN check is used to check whether the PSN of the message is equal to the ePSN of the corresponding QP; therefore, the request message check module needs to maintain the ePSN value of each QP;

R_Key check is used to check whether the operation type and space size required to access the MR when processing RDMA Write or RDMA Read request messages are within the access rights of the R_Key. The access rights of the R_Key can be obtained from the MRC.

Q_Key check, used to check whether the Q_Key of the request message matches the corresponding destination QP;

Message length check, including checking whether the message payload length matches the MTU and Opcode, and whether the total length of all message payloads in the RDMA Write request matches the DMA Length field in its RETH (RDMA Extended Transport Header). Therefore, the request message check module needs to record the RETH in the RDMA Write request message based on the QP, as well as the cumulative value of the payload length of all messages in the request.

Pad Count check is used to check whether the Pad Count field in BTH matches the Opcode.

When performing preliminary verification on the message, QP-related status information will be maintained. If an error is found during the verification process, the message will be marked as abnormal and the QP-related status information will be updated to facilitate decision-making on how to handle subsequent messages of the corresponding QP.

As shown in FIG2 , preferably, the first QP state module is further configured to process the request message according to the RC type QP hardware state when the QP type in the first QP state information is the RC type;

The first QP state module includes: a first state switching submodule;

A first state switching submodule, specifically used to switch to a normal state when the host creates or restarts a QP;

If the rationality check module determines that the result is no, it switches to the wait_send_nack state;

If the verification module notification request message is completed and it is a Class B error defined by the IB protocol, it switches to the wait_epsn state;

If the request message triggers a Class A or Class C error defined in the IB protocol, or a host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, it switches to the error state.

In actual use, when the QP type in the first QP state information is the RC type, four state transitions are performed, and the corresponding request message is processed in each state to further decide the message processing method. The QP hardware state includes four states: normal, wait_send_nack, wait_epsn, and error. When the host creates or restarts the QP, it enters the normal state, which is used for normal pipeline processing of request messages; in this state, if a message with an invalid PSN is detected, it enters the wait_send_nack state; if it is detected that the completion check module notifies a Class B error defined in the IB protocol, it enters the wait_epsn state; if it is detected that the request message being processed by this module triggers a Class A or Class C error defined in the IB protocol, the host configures the QP to enter an error-related state, and the completion check module notifies a Class A or Class C error defined in the IB protocol, it enters the error state;

The wait_send_nack state is used to wait for the network card to send the corresponding Unacknowledge message after discovering an invalid PSN message. In this state, if it is detected that the completion check module notifies the B-type error defined in the IB protocol, it will enter the wait_epsn state. If it is detected that the host configuration QP enters the error-related state and the completion check module notifies the A or C-type error defined in the IB protocol, it will enter the error state.

The wait_epsn state is used to wait for the peer end to resend the correct request message after the network card sends the Unacknowledge message caused by the Class B error. In this state, if a message with all the checks including the PSN check is detected to be correct, it will enter the normal state. If it is detected that the request message being processed by this module has a correct PSN but triggers the Class A or Class C error defined by the IB protocol, or the host configuration QP enters the error-related state, it will enter the error state.

The error state is used when the network card discards the corresponding QP message. In this state, the host can enter the normal state only after creating or restarting the QP.

Preferably, the first QP state module is further configured to:

When in error state, the network card is called to silently discard all request messages;

When in the normal state, the network card is called to discard the request message that is retransmitted with error, and the remaining request messages are marked according to the IB protocol based on the judgment result of the rationality check module;

When in the wait_send_nack state, the network card is called to process the correctly retransmitted request messages and the remaining request messages are discarded;

When in the wait_epsn state, the network card is called to process the request messages whose judgment result is yes or correctly retransmitted by the rationality check module, and the remaining request messages are discarded. Among them, the processing method of the request messages whose judgment result is yes by the rationality check module is consistent with the processing method in the normal state.

During actual use, the first QP state module operates as follows by calling the message processing method based on the QP state of the network card: if it is in the error state, all messages are silently discarded directly; if it is in the normal state, the erroneous retransmission message is silently discarded, and other messages are marked and processed according to the message verification results in accordance with the IB protocol requirements, such as marking as invalid request error, remote access error, PSN sequence error and other A/B/C errors defined by the IB protocol; if it is in the wait_send_nack state, only the correct retransmission message needs to be processed, and all other messages are discarded; if it is in the wait_epsn state, only the PSN correct message or the correct retransmission message needs to be processed, and the processing method of the PSN correct message is the same as that of the normal state, and all other messages are discarded.

Preferably, the RQE processing verification module includes: a cache module, an RQE state verification module, an RQE space verification module, a second screening module and a second QP state module;

The cache module is used to obtain the RQE of the active QP from the host and cache it to the network card in different QPs;

RQE status check module, used to check whether the RQE is correct, if not, it is marked as a malformed RQE;

The RQE space check module is used to check whether there is an RQE corresponding to the QP cached in the network card according to the request message. If so, the RQE corresponding to the QP cached is allocated to the request message. If not, an RNR error is triggered and the request message is marked.

The second screening module is used to screen the request messages whose judgment results of the RQE state verification module and the RQE space verification module are both yes;

The second QP state module is used to mark the request message as abnormal and update the second QP state information when either the judgment result of the RQE state check module or the judgment result of the RQE space check module is negative.

In actual use, when performing RQE processing, the RQE of the active QP will be obtained from the host in advance and cached to the network card, and the correctness of the RQE will be checked based on MRC and other information; when a request message that needs to consume RQE is received, if the corresponding QP has RQE, it can be directly taken out from the cache for use, if there is no RQE, the corresponding exception handling operation will be performed; in addition, when the corresponding QP is in an error state, the RQE processing and verification module will also actively read the RQE of the corresponding QP to facilitate the execution of the RQE refresh error completion operation. When judging whether the RQE space is sufficient to be allocated to the message, that is, judging whether the remaining host memory space corresponding to the RQE is sufficient to support the writing of the current message payload data, the correctness of the RQE will also be checked. If the RQE is correct and sufficient, the processing of the subsequent modules will continue, otherwise the corresponding exception handling operation will be performed. When performing the above RQE processing and judging whether the RQE space is sufficient to be allocated to the message, the corresponding QP status information will be maintained. When any abnormality is found when processing the message, the message will be marked as abnormal, and the QP status information will be updated at the same time to facilitate the decision-making of the corresponding QP subsequent message or RQE processing method.

In this embodiment, the RQE of the active QP is obtained from the host in advance and cached to the network card in different QPs. At the same time, the RQE is checked to see if it is correct based on the SGE (Scatter/Gather Elements), L_Key, MRC and other information in the RQE. If the RQE is wrong, it is marked as a malformed RQE for caching. When the RQE space check module receives a request message that needs to consume RQE, it checks whether the network card cache has the RQE of the corresponding QP. If there is an RQE, it is taken out and assigned to this request message. If not, it may trigger an RNR (Receiver Not Ready) error. The message processing method needs to be jointly decided with the status of the message being marked and the QP status and other information.

As shown in FIG3 , preferably, the second QP state module is further configured to process the request message according to the RC type QP hardware state when the QP type in the second QP state information is the RC type;

The second QP state module includes: a second state switching submodule;

The second state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the RQE space check module determines that the result is no or the request message has been marked as a Class B error defined by the IB protocol, it switches to the wait_send_nack state;

If the verification module notification request message is completed and it is a Class B error defined by the IB protocol, it switches to the wait_epsn state;

If the request message has been marked as a Class A or Class C error defined in the IB protocol, or a host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, it switches to the error state.

In actual use, when the QP type in the second QP state information is the RC type, four states are converted, and the corresponding request message is processed in each state to further decide the message processing method. When the host creates or restarts the QP, it enters the normal state, which is used for normal pipeline processing of request messages; in this state, if a message marked as normal but triggering an RNR error or a message marked as a Class B error defined by the IB protocol is detected, it enters the wait_send_nack state, if it is detected that the completion check module notifies the Class B error defined by the IB protocol, it enters the wait_epsn state, if it is detected that the request message being processed has been marked as a Class A or Class C error defined by the IB protocol, the host configures the QP to enter the error-related state, and the completion check module notifies the Class A or Class C error defined by the IB protocol, it enters the error state;

The wait_send_nack state is used to wait for the network card to send the corresponding Unacknowledge message after detecting the RNR error. In this state, if it is detected that the completion check module notifies the B-type error defined in the IB protocol, it will enter the wait_epsn state. If it is detected that the host configuration QP enters the error-related state and the completion check module notifies the A or C-type error defined in the IB protocol, it will enter the error state.

The wait_epsn state is used to wait for the peer end to resend the correct request message after the network card sends the Unacknowledge message caused by the Class B error. In this state, if a message marked as normal and with correct RNR verification is detected, the normal state is entered. If a request message is detected to be marked as a Class A or Class C error defined by the IB protocol, or the host configuration QP enters an error-related state, the error state is entered. If a message marked as normal but triggering an RNR error or a message marked as a Class B error defined by the IB protocol is detected, the wait_send_nack state is entered.

The error state is used when the network card discards the corresponding QP message and completes the RQE refresh error. In this state, the host can enter the normal state only after creating or restarting the QP.

Preferably, the second QP state module is further configured to:

When in error state, the network card is called to silently discard all request messages;

When in the wait_send_nack state, the network card is called to process the correctly retransmitted request messages and the remaining request messages are discarded;

When in the normal state or wait_epsn state, if the request message triggers an RNR error and the request message has been marked as a Class B error with a negative result of the rationality check module, the request message is re-marked as an RNR error.

In actual use, the second QP state module operates as follows by calling the message processing method based on the QP state of the network card: when the QP is in the normal state or the wait_epsn state, if the message triggers the RNR error and the message has been marked as a Class B error of a PSN sequence error by the request message verification module, the message is re-marked as an RNR error; if the message does not trigger the RNR error or has been marked as other Class A or Class C errors by the request message verification module, there is no need to modify the processing method decided by the request message verification module;

When QP is in the wait_send_nack state, only retransmitted packets marked as normal can be processed later, and other packets will be discarded;

When the QP is in the error state, all packets are discarded.

Preferably, the RQE space check module further comprises: a double check submodule;

The double check submodule is used to check whether the RQE corresponding to the QP is deformed and whether the remaining host memory space corresponding to the RQE supports the writing of the current request message payload data after the RQE corresponding to the QP already exists in the network card;

If the RQE is marked as malformed, the request message is marked as a malformed RQE error;

If the remaining host memory space corresponding to the RQE does not support the writing of the current request message payload data, the request message is marked as an invalid request error.

In actual use, when processing a message based on the read RQE, it will also check whether the RQE is malformed and whether the remaining host memory space corresponding to the RQE is sufficient to support the writing of the current message payload data: if the RQE is found to be marked as malformed, the message is marked as a malformed RQE error; if the remaining host memory space corresponding to the RQE is insufficient to write the current message payload data, the message is marked as an invalid request error. The above two errors are both Class A or Class C errors defined by the IB protocol.

Preferably, the second QP state module further includes: a third state switching submodule;

The third state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the request message has been marked as a Class A or Class C error defined by the IB protocol, or a host configuration QP error, or the completion check module notifies a Class A or Class C error defined by the IB protocol, or the request message is marked as a malformed RQE error or an invalid request error, it switches to the error state.

In actual application, when the QP type in the second QP state information is the RC type, two state conversions are performed, and the request message is processed in each state to further decide the processing method of the message: when processing the message based on the read RQE, in addition to recording the remaining host memory space size corresponding to the above RQE based on the QP, it is also necessary to maintain the QP hardware state for the RC type QP, which includes two states: normal and error. When the host creates or restarts the QP, it enters the normal state, which is used for normal pipeline processing of request messages; in this state, if it is detected that the request message being processed has been marked as a Class A or Class C error defined by the IB protocol, the request message being processed is marked as the above two errors in this module, the host configures the QP to enter the error-related state, and the completion verification module notifies the Class A or Class C error defined by the IB protocol, any of the four situations, then enter the error state; the error state is used for the network card to discard the message corresponding to the QP; in this state, if the host creates or restarts the QP, it can enter the normal state.

As shown in FIG4 , preferably, the request message load upload module includes: a load upload module, a load data writing module and a third QP state module;

The load upload module is used to determine whether the request message needs to be loaded.

The load data writing module is used to obtain the corresponding physical page address from the host according to the remote address in the request message or the address in the RQE when the load upload module judges that the result is yes, write the request message load into the host memory according to the physical page address, and construct a request message descriptor;

The third QP state module is used to process the request message according to the RC type QP hardware state when the QP type in the third QP state information is the RC type.

During actual use, when the request message payload upload module writes the message payload into the host memory, it first determines whether the message needs to be uploaded. If not, it directly processes the subsequent modules. If necessary, it obtains the corresponding physical page address from the host based on the remote address in the request message or the address in RQE, and then writes the message payload data into the host memory according to the physical page address. Finally, it constructs a request message descriptor based on the processing status of the message and sends it to the completion verification module.

In this embodiment, when the first message of the request for load upload is received, the remote access address and length in the message RETH or the SGE address and length in the allocated RQE are recorded, and then the physical page addresses required for the corresponding requests are batch read and cached to the network card, so that subsequent messages of the request can directly obtain the corresponding physical page addresses from the cache, thereby reducing PCIE bandwidth occupancy and subsequent message processing delays.

Preferably, the third QP state module comprises: a fourth state switching submodule;

The fourth state switching submodule is specifically used to switch to the normal state when the host creates or restarts the QP;

If the request message has been marked as a Class A or Class C error defined in the IB protocol, or a host configuration QP error, or the completion check module notifies any of the Class A or Class C errors defined in the IB protocol, it switches to the error state.

In actual application, it is necessary to maintain the access address and remaining length of the next message of the current request based on QP, so as to allocate the corresponding memory cache space for the payload data of the subsequent messages of the request. In addition, the request message payload upload module does not verify the message, but in order to ensure the consistency of pipeline processing messages, it is still necessary to maintain the QP hardware state for RC type QP, which includes two states: normal and error. Among them: normal state is used for normal pipeline processing of request messages. When the host creates or restarts QP, it enters the normal state; in this state, if it is detected that the request message being processed has been marked as a Class A or Class C error defined by the IB protocol, the host configures QP to enter the error-related state, and the completion verification module notifies the Class A or Class C error defined by the IB protocol, it enters the error state; error state is used for the network card to discard the corresponding QP message; in this state, if the host creates or restarts QP, it can enter the normal state.

As shown in FIG5 , preferably, the verification module is completed, including: a request message processing mode determination module and a notification module;

A request message processing mode determining module, configured to determine a request message processing mode according to the first QP state information, the second QP state information and the third QP state information included in the request message descriptor;

The notification module is used to send the message information in the request message descriptor to the request message verification module, the RQE processing verification module and the request message load upload module respectively to update the corresponding QP state information.

In actual use, the completion verification module determines the processing of the message by the previous modules according to the information recorded in the request message descriptor, which is specifically reflected as the first QP status information, the second QP status information and the third QP status information, and then decides how to construct the response message descriptor and generate CQE based on the IB protocol. Among them, the response message includes Acknowledge message, Unacknowledge message and Read Response message. When constructing the descriptor of the Unacknowledge message, the QPN (Queue Pair Number), error type, ePSN and other information in the corresponding request message descriptor will be notified to the request message verification module, the RQE processing verification module, and the request message load upload module, so that these three modules can update the status information of the corresponding QP.

Preferably, the response message construction module is specifically used to generate a response message descriptor according to the request message descriptor, and construct a response message according to the response message descriptor;

If the response message is a read response message, the response message construction module is further specifically used to construct the response message according to the response message descriptor, the remote address in the request message, and the physical page address.

In actual application, the construction of Acknowledge and Unacknowledge messages can be done directly based on descriptor information, QPC and other information; as for the construction of Read Response messages, data needs to be obtained from the host to fill the message payload. The processing flow of constructing Read Response messages is divided into three steps: physical page address acquisition, MR data reading, and message payload data filling and encapsulation; among them: the physical page address acquisition step is used to obtain the physical page address of the access space required by the RDMA read request from the host memory according to the remote address of the Read request, R_Key and MRC of the network card cache; the MR data acquisition step is used to read the data of the MR corresponding space from the host memory again according to the physical page address; the message payload data filling and encapsulation step is used to fill the read MR data into the message payload field, and then construct the message header for message encapsulation and forwarding.

In the embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

In addition, all functional modules in the embodiments of the present invention may be integrated into one processor, or each module may be a separate device, or two or more modules may be integrated into one device; each functional module in the embodiments of the present invention may be implemented in the form of hardware or in the form of hardware plus software functional units.

It should be understood that the use of "system", "device", "unit" and/or "module" in this application is only a method for distinguishing different components, elements, parts, parts or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and/or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

The above is a detailed description of a message processing system provided by the present invention. The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features disclosed herein.

Claims (11)

1. A message processing system, comprising:

The request message checking module is used for checking the received request message and maintaining the first QP state information, and screening out the request message passing the checking;

The RQE processing and checking module is used for checking the RQE acquired from the host and maintaining second QP state information, and checking whether the space of the RQE is enough to be distributed to the request message for use;

The request message load uploading module is used for writing the request message load into the host computer and maintaining third QP state information according to the remote address in the request message or the address in the RQE;

the completion checking module is used for determining a request message processing mode according to the first QP state information, the second QP state information and the third QP state information and notifying the request message processing mode to maintain synchronization of the first QP state information, the second QP state information and the third QP state information;

the response message construction module is used for constructing a response message according to the request message processing mode;

The request message checking module comprises a sending checking module, a rationality checking module, a first screening module and a first QP state module, wherein the first QP state module comprises a first state switching sub-module, is used for switching to a normal state when a host creates or restarts QP, is switched to a wait_send_nack state if the rationality checking module judges that the request message is not in the normal state, is switched to a wait_ epsn state if the completion checking module informs that the request message is a B-type error defined by IB protocol, is switched to an error_ epsn state if the request message triggers an A-type or C-type error defined by IB protocol or the host configures QP error or is switched to an error when the completion checking module informs any one of the A-type or C-type errors defined by IB protocol;

When the request message is in an error state, the network card is called to directly silence and discard all the request messages, when the request message is in a normal state, the network card is called to discard the request messages which are retransmitted in error, the rest of the request messages are marked according to the judging result of the rationality checking module according to an IB protocol, when the request message is in a wait_send_nack state, the network card is called to process the request messages which are retransmitted correctly, the rest of the request messages are discarded, when the request message is in a wait_ epsn state, the network card is called to process the request messages which are judged to be yes or retransmitted correctly, the rest of the request messages are discarded, wherein the processing mode of the request messages which are judged to be yes by the rationality checking module is consistent with the processing mode when the request messages are in the normal state.

2. The message processing system of claim 1, wherein,

The sending verification module is used for verifying whether the received request message is the request message of the message processing system;

The rationality checking module is used for checking whether the request message is rational;

The first screening module is used for screening the request message of which the judgment result of the sending verification module and the judgment result of the rationality verification module are both yes;

The first QP state module is configured to mark that the request packet is abnormal and update the first QP state information when either the determination result of the sending verification module and the determination result of the rationality verification module are negative;

And the first QP state module is further configured to correspondingly process the request packet according to an RC type QP hardware state when the QP type in the first QP state information is an RC type.

3. The message processing system of claim 2, wherein the RQE processing and checking module comprises a buffer module, a RQE state checking module, a RQE space checking module, a second screening module, and a second QP state module;

the caching module is used for acquiring the RQE of the active QP from the host and caching the RQE to the network card according to the QP;

the RQE state checking module is used for checking whether the RQE is correct, if not, marking the RQE as malformed;

The RQE space verification module is used for checking whether the RQE of the cached corresponding QP exists in the network card according to the request message, if so, distributing the RQE of the cached corresponding QP to the request message, and if not, triggering RNR error and marking the request message;

the second screening module is configured to screen the request packet in which the RQE state verification module and the RQE space verification module both determine that the result is positive;

and the second QP state module is configured to mark that the request packet is abnormal and update the second QP state information when the RQE state checking module determines that either the result or the RQE space checking module determines that the result is negative.

4. The message processing system of claim 3, wherein,

The second QP state module is further configured to correspondingly process the request packet according to an RC type QP hardware state when the QP type in the second QP state information is an RC type;

the second QP state module comprises a second state switching sub-module;

The second state switching sub-module is specifically configured to switch to a normal state when the host creates or restarts the QP;

If the RQE space verification module judges that the result is no or the request message is marked as the B-type error defined by IB protocol, switching to wait_send_nack state;

if the completion checking module informs that the request message is a B-type error defined by an IB protocol, switching to a wait_ epsn state;

And if the request message is marked as an A-type or C-type error defined by IB protocol or the host configures QP error or the completion checking module informs any one of the A-type or C-type errors defined by IB protocol, switching to error state.

5. The message processing system of claim 4, wherein,

The second QP state module is further used for,

When the request message is in an error state, calling a network card to directly silence and discard all the request messages;

when the request message is in the wait_send_nack state, calling a network card to process the correctly retransmitted request message, and discarding the rest request messages;

And when the request message is in a normal state or a wait_ epsn state, if the request message triggers an RNR error and the request message is marked as a B-type error with a negative judgment result of the rationality checking module, the request message is re-marked as the RNR error.

6. The message processing system of claim 3, wherein the RQE space verification module further comprises a double verification sub-module;

The double checking sub-module is used for checking whether the RQE is malformed and whether the residual host memory space corresponding to the RQE supports the current writing of the load data of the request message after the RQE corresponding to the QP exists in the network card;

if the RQE is marked as malformed, marking the request message as malformed RQE error;

And if the residual host memory space corresponding to the RQE does not support the current load data writing of the request message, marking the request message as an invalid request error.

7. The message processing system of claim 6, wherein the second QP status module further comprises a third status switch sub-module;

the third state switching sub-module is specifically configured to switch to a normal state when the host creates or restarts the QP;

And switching to an error state if the request message is marked as either a class A or class C error defined by IB protocol or the host configuration QP error or the completion checking module notifies a class A or class C error defined by IB protocol or the request message is marked as either the malformed RQE error or the invalid request error.

8. The message processing system of claim 1, wherein the request message load upload module comprises a load upload module, a load data write module, and a third QP status module;

The load uploading module is used for judging whether the request message needs load uploading or not;

The load data writing module is used for acquiring a corresponding physical page address from a host according to a remote address in the request message or an address in the RQE when the load uploading module judges that the load data is yes, writing the load of the request message into the main memory according to the physical page address, and constructing a request message descriptor;

and the third QP state module is configured to correspondingly process the request packet according to an RC type QP hardware state when the QP type in the third QP state information is an RC type.

9. The message processing system of claim 8, wherein the third QP status module comprises a fourth status switch sub-module;

a fourth state switching sub-module, specifically configured to switch to a normal state when the host creates or restarts the QP;

And if the request message is marked as a class A or class C error defined by the IB protocol or a host configuration QP error or the completion checking module informs any one of the class A or class C errors defined by the IB protocol, switching to an error state.

10. The message processing system of claim 8, wherein the completion verification module comprises a request message processing mode determination module and an announcement module;

The request message processing mode determining module is configured to determine a request message processing mode according to the first QP state information, the second QP state information, and the third QP state information included in the request message descriptor;

And the notification module is used for respectively sending the message information in the request message descriptor to the request message verification module, the RQE processing verification module and the request message load uploading module so as to update the corresponding QP state information.

11. The message processing system of claim 8, wherein the response message construction module is specifically configured to generate a response message descriptor according to the request message descriptor, and construct a response message according to the response message descriptor;

if the response message is a read response message, the response message construction module is further specifically configured to construct the response message according to the response message descriptor, the remote address in the request message, and the physical page address.