CN118316778B - UAR page allocation method and device for IB network card - Google Patents
UAR page allocation method and device for IB network card Download PDFInfo
- Publication number
- CN118316778B CN118316778B CN202410417373.2A CN202410417373A CN118316778B CN 118316778 B CN118316778 B CN 118316778B CN 202410417373 A CN202410417373 A CN 202410417373A CN 118316778 B CN118316778 B CN 118316778B
- Authority
- CN
- China
- Prior art keywords
- uar
- page
- linked list
- space
- pages
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000013507 mapping Methods 0.000 claims abstract description 5
- 238000010586 diagram Methods 0.000 description 6
- 239000002699 waste material Substances 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000003780 insertion Methods 0.000 description 3
- 230000037431 insertion Effects 0.000 description 3
- 235000009781 Myrtillocactus geometrizans Nutrition 0.000 description 2
- 240000009125 Myrtillocactus geometrizans Species 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000007417 hierarchical cluster analysis Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
- H04L41/0253—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols using browsers or web-pages for accessing management information
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a UAR page allocation method and device for an IB network card, wherein the method comprises the steps of dividing a PCIE base address register space into UAR pages, mapping addresses of the UAR pages into corresponding host side starting addresses by utilizing offsets of PCIE base addresses, initializing a global linked list, adding the starting addresses corresponding to each UAR page into the global linked list, taking UAR pages out of the global linked list when a queue pair QP is created, and allocating idle WQE spaces of the UAR pages to the queue pairs so that each queue pair can obtain independent PCIE base address register space. The technical scheme of the invention avoids UAR page resource contention caused by concurrency of multiple QPs, and improves the performance of the network card.
Description
Technical Field
The invention belongs to the field of network communication, and particularly relates to a UAR page allocation method and device for an IB network card.
Background
RDMA (Remote Direct Memory Access), remote direct memory access), is an efficient network communication technology that allows direct exchange of memory data between computers without going through an operating system. A key advantage of RDMA is that it reduces latency and CPU load in the data transfer process. BlueFlame is an important supplement to conventional RDMA technology, and improves the efficiency and performance of overall network communication by optimizing memory write operations, allows network card hardware to directly process operations of constructing and processing network packets, provides a more efficient way to process RDMA memory write operations, reduces dependence on CPU, and thereby reduces delay and improves performance. UAR (User ACCESS REGIN, user direct access region) pages (UAR pages) are regions of memory that are used to store control information necessary for RDMA operations. UAR pages allow HCAs (high performance network cards) to efficiently access these control information to perform network operations. Each UAR page contains a plurality of UARs, each UAR corresponding to a particular network operation or queue. BlueFlame and UAR pages work together, so that the efficiency of data packet processing is improved, the quick access of control information required by operation is ensured, the RDMA process is optimized together, the delay is reduced, and the throughput is improved.
However Blueframe WQE (Work Queue Element, work queue entry for specifying and controlling RDMA operations) are aligned in 64 bytes, so if submitted directly through UAR, the host side needs to make one copy, in a multi-threaded or multi-process environment, if multiple threads or processes share the same QP and UAR Page, they may attempt to write to UAR Page at the same time, resulting in race conditions and performance problems.
The current mellanox IB network card considers the requirements of resource management and performance optimization in design, and allocates 16 UAR pages in each context (context), as shown in fig. 1. Where the context contains data structures and information associated with a particular IB device or resource. This can prevent race condition problems from occurring with the increase of context. However, when a large number of QPs (Queue Pair) are created in the same context, the concurrency of these QPs causes race conditions to occur, thereby affecting performance.
Disclosure of Invention
The invention aims to provide a UAR page allocation method and device for an IB network card, aiming at improving the network card performance under the concurrent condition of multiple QPs.
According to a first aspect of the present invention, there is provided a UAR page allocation method for an IB network card, including:
dividing a PCIE base address register space into UAR pages, mapping addresses of the UAR pages into corresponding host side starting addresses by utilizing offsets of PCIE base addresses, initializing a global linked list, and adding the starting addresses corresponding to each UAR page into the global linked list;
When creating a queue pair QP, taking UAR pages out of the global linked list, and distributing the free WQE space of the UAR pages to the queue pairs, so that each queue pair obtains independent PCIE base address register space.
Preferably, the adding each UAR page to the global linked list further includes:
and sequentially inserting the starting address corresponding to each UAR page into the tail part of the global linked list.
The number of UAR pages is independent of process or thread context.
The retrieving the UAR page from the global linked list further includes:
And taking out the current UAR page from the head of the global linked list, deleting the current UAR page from the head of the linked list, and then adding the current UAR page to the tail of the global linked list.
The allocating the free WQE space of the UAR page to the queue pair further includes:
If the current UAR page has an idle WQE space, the idle WQE space is distributed to the current queue pair;
If the current UAR page does not have the free WQE space, continuing to take out the subsequent UAR page from the global linked list until the free WQE space exists in the subsequent UAR page, and distributing the free WQE space to the current queue pair.
According to a second aspect of the present invention, there is provided a UAR page allocating apparatus for an IB network card, comprising:
The PCIE base address register space is divided into UAR pages, addresses of the UAR pages are mapped into corresponding host side starting addresses by utilizing the offset of the PCIE base address, a global linked list is initialized, and the starting address corresponding to each UAR page is added into the global linked list;
And the page allocation unit is used for taking UAR pages out of the global linked list when the queue pair QP is created, and allocating the free WQE space of the UAR pages to the queue pairs so that each queue pair obtains independent PCIE base address register space.
Compared with the related art, the technical scheme of the invention has the following advantages:
UAR pages are managed by using a global linked list mode, the number limit of UAR pages owned by one process is relieved, the whole BAR0 space is utilized, and the situation that race conditions are not generated when all QPs in one context are concurrent can be ensured, so that the software performance is ensured, the access conflict of a plurality of QPs to the BAR is avoided, each QP can access an independent BAR, all QPs in one process or thread can monopolize the resources of the BAR space, and the BAR space resource waste is reduced. When all QPs in one process or thread perform concurrent operation, lock-free operation can be realized at the software level, and CPU resource waste caused by lock competition of the CPU is reduced. Since all QPs monopolize doorbell regions in UAR pages, performance bottlenecks caused by UAR page resource contention due to concurrency of multiple QPs are avoided.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure and process particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the related art, the drawings that are required to be used in the embodiments or the related technical descriptions will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to the drawings without any inventive effort for a person skilled in the art.
Fig. 1 is a schematic diagram of UAR page allocation according to the related art.
Fig. 2 is a flow chart of a UAR page allocation method according to the invention.
Fig. 3 is a schematic diagram of splitting a PCIE BAR into UAR pages according to the present invention.
FIG. 4 is a diagram of a UARpage global linked list in accordance with the present invention.
FIG. 5 is a schematic diagram of a UAR page global linked list fetch and join process in accordance with the present invention.
FIG. 6 is a diagram of allocation states of WQE space and QP according to the present invention.
Fig. 7 is a schematic diagram of UAR page allocation situation according to the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which are derived by a person skilled in the art from the embodiments according to the invention without creative efforts, fall within the protection scope of the invention.
Based on the above analysis, the present invention proposes a UAR page allocation method and apparatus for IB network card, which removes the restriction of PCIE BAR (Base ADDRESS REGISTER ) space on QP in the drive, and does not classify UAR pages in BAR space, so that all QPs in each context can use only one doorbell area in UAR page, i.e. space of WQE.
Referring to the flowchart of fig. 2, the UAR page allocation method for the IB network card provided by the invention includes:
Step 101, dividing a PCIE base address register space into UAR pages, mapping addresses of the UAR pages into corresponding host side starting addresses by utilizing offsets of PCIE base addresses, initializing a global linked list, and adding the starting addresses corresponding to each UAR page into the global linked list.
As shown in FIG. 3, the 32M space of BAR0 is first split into UAR pages of 4K size in their entirety. Then creating a linked list UAR_Page_1ist of the UAR page of the global type at the host side, wherein the UAR page adopts a strategy of dynamic double-end queue management. The strategy employs a dynamically resized double-ended queue (uar_page_list) that supports efficient insert and delete operations at both ends. This feature of a double-ended queue allows it to be used as a stack (stack) as well as a queue (queue), providing great flexibility. In the invention, UAR pages are added by adopting a tail insertion strategy, and new UAR pages are always added to the tail of the linked list. Tail insertion ensures UARpage sequential so that the newly added UARpage is always in the last position of the linked list. Maintenance and expansion can be achieved with the flexibility of double-ended queues. Assuming that the UARs of the current BAR space are all used, a tail delete and head reuse strategy may be used, reusing one UAR page.
The UAR pages are converted into addresses which can be directly accessed by a host computer in an address mapping mode, the addresses of all UAR pages allocated in the BAR0 space are stored in a UAR_page_list of a global linked list of UAR pages, the UAR pages are sequentially added to UAR pages 8192 from UAR page0, and the addresses of each UAR page are sequentially inserted into the tail of the linked list, as shown in fig. 4.
Since one global linked list is used to maintain all UAR pages, it is independent of context for each thread or process.
And 102, when a queue pair QP is created, taking UAR pages out of the global linked list, and distributing the free WQE space of the UAR pages to the queue pairs so that each queue pair obtains independent PCIE base address register space.
When a QP is first created, one UAR page is fetched from the head of the linked list, the fetched UAR page is assigned to the QP, and the fetched UAR page is deleted from the head of the linked list and then added to the tail of the linked list, as shown in fig. 5. A head extraction policy is employed when fetching UAR pages, which corresponds to tail insertion, which allows elements to be quickly removed from the head of the linked list. The time complexity of this operation is O (1), which can be done in a constant time, guaranteeing an efficient access speed.
The QP created by each context obtains UAR pages from the linked list, and there is no limit to the number of UAR pages in one context. One UAR page contains 4 WQE spaces, each available for one QP, as shown in fig. 6.
In an alternative embodiment, if the current UAR page does not have free WQE space, indicating that all WQEs in current UARpage are allocated to previous QP, then the subsequent UAR page is continued to be fetched from the global linked list until free WQE space is found and allocated to the current QP.
Therefore, referring to fig. 7, up to 32K QPs applied in one process can independently use doorbell regions in their UAR pages, so that performance influence caused by sharing one UARpage among QPs in one process can be effectively avoided.
As described above, in the related art, a process includes at most 16 UAR pages, so that more than 64 QPs are concurrent in one process, a race condition occurs, which affects performance. Compared with the related art, the UAR page allocation method for the IB network card uses the global linked list mode to manage UAR pages, so that the quantity limit of UAR pages owned by one process is relieved, the whole BAR0 space is fully utilized, race conditions can not be generated when all QPs in one context are concurrent, the software performance is ensured, access conflict of a plurality of QPs to the BAR is avoided, each QP can access an independent BAR, all QPs in one process or thread can monopolize the resources of the BAR space, and the BAR space resource waste is reduced. When all QPs in one process or thread perform concurrent operation, lock-free operation can be realized at the software level, and CpU resource waste caused by lock competition of the CPU is reduced. Since all QPs monopolize doorbell regions in UAR pages, performance bottlenecks caused by UAR page resource contention due to concurrency of multiple QPs are avoided.
Accordingly, the present invention provides in a second aspect a UAR page allocation apparatus for an IB network card, comprising:
The PCIE base address register space is divided into UAR pages, addresses of the UAR pages are mapped into corresponding host side starting addresses by utilizing the offset of the PCIE base address, a global linked list is initialized, and the starting address corresponding to each UAR page is added into the global linked list;
And the page allocation unit is used for taking UAR pages out of the global linked list when the queue pair QP is created, and allocating the free WQE space of the UAR pages to the queue pairs so that each queue pair obtains independent PCIE base address register space.
The above device may be implemented by the UAR page allocation method for an IB network card provided by the embodiment of the first aspect, and specific implementation manner may be referred to the description in the embodiment of the first aspect, which is not repeated herein.
It is understood that the storage structures, names and parameters described in the above embodiments are only examples. Those skilled in the art may also make and adjust the structural features of the above embodiments as desired without limiting the inventive concept to the specific details of the examples described above.
Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that modifications may be made to the technical solutions described in the foregoing embodiments or equivalents may be substituted for some of the technical features thereof, and these modifications or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention in essence.
Claims (8)
1. A UAR page allocation method for an IB network card, comprising:
Dividing a PCIE base address register space into UAR pages, mapping addresses of the UAR pages into corresponding host side starting addresses by utilizing offsets of PCIE base addresses, initializing a global linked list, and adding the starting addresses corresponding to each UAR page into the global linked list;
when a queue pair QP is created, a UAR page is taken out from the global linked list, and an idle WQE space of the UAR page is distributed to the queue pairs, so that each queue pair obtains an independent PCIE base address register space;
the allocating the free WQE space of the UAR page to the queue pair further includes:
If the current UAR page has an idle WQE space, the idle WQE space is distributed to the current queue pair;
If the current UAR page does not have the free WQE space, continuing to take out the subsequent UAR page from the global linked list until the free WQE space exists in the subsequent UAR page, and distributing the free WQE space to the current queue pair.
2. The UAR page allocation method for an IB network card according to claim 1, wherein adding a start address corresponding to each UAR page to the global linked list further comprises:
and sequentially inserting the starting address corresponding to each UAR page into the tail part of the global linked list.
3. The UAR page allocation method for an IB network card of claim 1, wherein the number of UAR pages is independent of process or thread context.
4. The UAR page allocation method for an IB network card of claim 1, wherein said retrieving a UAR page from said global linked list further comprises:
And taking out the current UAR page from the head of the global linked list, deleting the current UAR page from the head of the linked list, and then adding the current UAR page to the tail of the global linked list.
5. A UAR page allocation apparatus for an IB network card, comprising:
The PCIE base address register space is divided into UAR pages, addresses of the UAR pages are mapped into corresponding host side starting addresses by utilizing the offset of the PCIE base address, a global linked list is initialized, and the starting address corresponding to each UAR page is added into the global linked list;
The page allocation unit is used for taking UAR pages out of the global linked list when a queue pair QP is created, and allocating the idle WQE space of the UAR pages to the queue pairs so that each queue pair obtains independent PCIE base address register space;
The page allocation unit is further configured to:
If the current UAR page has an idle WQE space, the idle WQE space is distributed to the current queue pair;
If the current UAR page does not have the free WQE space, continuing to take out the subsequent UAR page from the global linked list until the free WQE space exists in the subsequent UAR page, and distributing the free WQE space to the current queue pair.
6. The UAR page allocation apparatus for an IB network card as defined in claim 5, wherein said linked list generating unit is further configured to:
and sequentially inserting the starting address corresponding to each UAR page into the tail part of the global linked list.
7. The UAR page allocation apparatus for an IB network card of claim 5, wherein the number of UAR pages is independent of process or thread context.
8. The UAR page allocation apparatus for an IB network card according to claim 7, wherein said page allocation unit is further configured to:
And taking out the current UAR page from the head of the global linked list, deleting the current UAR page from the head of the linked list, and then adding the current UAR page to the tail of the global linked list.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410417373.2A CN118316778B (en) | 2024-04-08 | 2024-04-08 | UAR page allocation method and device for IB network card |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410417373.2A CN118316778B (en) | 2024-04-08 | 2024-04-08 | UAR page allocation method and device for IB network card |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118316778A CN118316778A (en) | 2024-07-09 |
| CN118316778B true CN118316778B (en) | 2024-12-13 |
Family
ID=91730680
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410417373.2A Active CN118316778B (en) | 2024-04-08 | 2024-04-08 | UAR page allocation method and device for IB network card |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118316778B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153785A (en) * | 2021-11-29 | 2022-03-08 | 北京志凌海纳科技有限公司 | Memory management method and device based on remote direct memory access |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7543290B2 (en) * | 2001-04-11 | 2009-06-02 | Mellanox Technologies Ltd. | Multiple queue pair access with single doorbell |
| US8051212B2 (en) * | 2001-04-11 | 2011-11-01 | Mellanox Technologies Ltd. | Network interface adapter with shared data send resources |
| CN105867945B (en) * | 2016-04-20 | 2018-11-16 | 华为技术有限公司 | A kind of BIOS starting method and device |
| CN112804372B (en) * | 2020-12-31 | 2023-03-24 | 武汉思普崚技术有限公司 | User terminal grouping method, system, device and storage medium |
| CN114816528B (en) * | 2022-04-30 | 2025-01-24 | 苏州浪潮智能科技有限公司 | A method, device, electronic device and medium for accessing a register |
| CN117724874B (en) * | 2024-02-06 | 2024-04-26 | 珠海星云智联科技有限公司 | Method, computer device and medium for managing shared receive queues |
-
2024
- 2024-04-08 CN CN202410417373.2A patent/CN118316778B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114153785A (en) * | 2021-11-29 | 2022-03-08 | 北京志凌海纳科技有限公司 | Memory management method and device based on remote direct memory access |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118316778A (en) | 2024-07-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113535633B (en) | On-chip caching device and read-write method | |
| US8549521B2 (en) | Virtual devices using a plurality of processors | |
| US9448857B2 (en) | Memory access method for parallel computing | |
| CN109690498B (en) | Memory management method and equipment | |
| US10209900B2 (en) | Buffer allocation and memory management using mapping table | |
| US9086920B2 (en) | Device for managing data buffers in a memory space divided into a plurality of memory elements | |
| US20080162834A1 (en) | Task Queue Management of Virtual Devices Using a Plurality of Processors | |
| US11989588B2 (en) | Shared memory management method and device | |
| EP3077914B1 (en) | System and method for managing and supporting virtual host bus adaptor (vhba) over infiniband (ib) and for supporting efficient buffer usage with a single external memory interface | |
| JP6584529B2 (en) | Method and apparatus for accessing a file and storage system | |
| CN110162395B (en) | Memory allocation method and device | |
| US7552232B2 (en) | Speculative method and system for rapid data communications | |
| WO2024036985A1 (en) | Storage system, computational storage processor and solid-state drive thereof, and data reading method and data writing method therefor | |
| WO2025139598A1 (en) | Physical memory expansion architecture and method for server, and server, device and medium | |
| CN116644002A (en) | Memory management method and device based on RDMA | |
| US7913059B2 (en) | Information processing device, data transfer method, and information storage medium | |
| CN115129621B (en) | Memory management method, device, medium and memory management module | |
| CN110445580B (en) | Data transmission method and device, storage medium, and electronic device | |
| CN118316778B (en) | UAR page allocation method and device for IB network card | |
| CN116483536B (en) | Data scheduling method, computing chip and electronic equipment | |
| CN115586943B (en) | Hardware marking implementation method for dirty pages of virtual machine of intelligent network card | |
| CN118819808A (en) | Memory allocation method, device, equipment, chip, storage medium and product | |
| CN120216151A (en) | Queue management method and queue manager implemented in UFS device | |
| US20160055112A1 (en) | Return available ppi credits command | |
| JP3849578B2 (en) | Communication control device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |