What about other headers?

Posted Jul 24, 2006 11:50 UTC (Mon) by christian.convey (guest, #39159)
Parent article: OLS: A proposal for a new networking API

I hope this question isn't born out of ignorance...

The DMA idea sounds great, but I'm curious how this works when protocol layers will want to add headers to the region of memory that exactly preceeds my application-level data.

For example, suppose I have application-data messages that are 256 byte long. So I request a 256-byte-long user-space DMA region, and it's mapped to my process' VM address range x10000000 - x100000FF. And I then populate all 256 bytes of that region with application-level data.

If the TCP and IP layers are going to bolt their headers onto the beginning of the data I'm sending, won't each of those layers (1) allocate a buffer big enough for that layer's header + the data from the higher protocol level, and then (2) copy the higher-level's data into that new buffer? If so, I don't see how zero-copy is achieved.

It seems to me like we'd almost need the application to announce the purpose for which it intends to use the DMA region, so that the allocator can include extra space at the beginning/end for the network stack to use.
For example (not ideal, but just to clarify my point):
int dma_alloc(AF_INET, SOCK_STREAM, dma_mem_t *handle, size_t size, int flags);

to post comments

What about other headers?

Posted Jul 24, 2006 15:55 UTC (Mon) by dlang (guest, #313) [Link] (4 responses)

not nessasarily, the kernel support scatter-gather internally which would let you identify the TCP header (in one area of memory) and the data (in another area or areas of memory) and tell the driver where they are, the driver will get all the pieces in one operation (this is even easier with an IOMMU, with that the kernel can program the address space so that the driver doesn't know that the pieces are seperated)

I don't know if this can be done with the userspace DMA work, but I'm not seeing a reason it couldnt'.

What about other headers?

Posted Jul 26, 2006 18:33 UTC (Wed) by mikov (guest, #33179) [Link] (3 responses)

Do all (most) Ethernet adapters support scatter-gather DMA ?

What about other headers?

Posted Jul 26, 2006 19:11 UTC (Wed) by dlang (guest, #313) [Link] (2 responses)

the higher-end cards support scatter-gather, but with an Opteron and it's IOMMU this functionality is available without the card needing to support it (the IOMMU remaps the address space so that the card thinks it's doing a old-fashiond single-area DMA, when it's really pulling/pushing the info to multiple sections of ram)

David Lang

What about other headers?

Posted Jul 27, 2006 1:11 UTC (Thu) by mikov (guest, #33179) [Link] (1 responses)

Thanks for pointing this - I must be falling behind the times because I didn't know about Opteron's IOMMU. I found a good description here: http://www.amd.com/us-en/assets/content_type/white_papers...

Is there equivalent functionality in Intel's chipsets (since they can't put it in the CPU) ? If not, it doesn't seem prudent to rely on it for zero-copy.

What about other headers?

Posted Jul 27, 2006 1:51 UTC (Thu) by dlang (guest, #313) [Link]

My understanding is that the Intel chips do not have an IOMMU at this time.

however this just means that the kernel and drivers have to fall back to the non-zero-copy mechanism if the hardware doesn't support scatter-gather nativly

it's a matter of useing it if it's there, and doing the best you can if it's not (and remember that if you compile the kernel for Opteron-only as opposed to K8 architecture you get more performance becouse you get to leave out the tests for the IOMMU and the code to handle it, so it doesn't always cost you :-)