Exposing storage devices as memory

Storage devices are in a period of extensive change. As they get faster and become byte-addressable by the CPU, they tend to look increasingly like ordinary memory. But they aren't memory, so it still isn't clear what the best model for accessing them should be. Adam Manzanares led a session during the memory-management track of the 2018 Linux Storage, Filesystem, and Memory-Management Summit, where his proposal of a new access mechanism ran into some skepticism.

As things stand now, he said, there are two ways of exposing a storage device as memory: adding it as a swap device, or allowing applications to directly map the device into their address space. But, as memory-like as these devices are, they still do not have the same access times as dynamic RAM and, despite improvements, they still do not have the same write endurance. So he is not sure that making them look like ordinary memory is the correct model.

He has a different idea: extend the virtual-memory idea to provide access to memory-like storage devices. Each device would have some ordinary RAM allocated for it; that memory would be used as a cache for data stored in the device itself. That device is represented in the kernel by a simple character device (/dev/umap) that would allow an application to map a portion of the storage device's space into its address space. The primary purpose of /dev/umap is to catch page faults; it can respond to them by moving data between the RAM cache and the storage device.

The developers in the room thought that this arrangement looked a lot like using the storage device as a swap area; Manzanares responded that he didn't want to use the existing swap mechanism because it depends on the block layer, which he thinks is expensive and wants to avoid. Just using the page cache also lacks appeal, he said; it is too memory-intensive and does not give applications enough control. As it is, he said, /dev/umap outperforms both the memory-mapping and swapping alternatives.

Dan Williams suggested that, rather than creating a completely new mechanism for accessing this memory, it would be better to start with the existing filesystem APIs and the page cache and, if necessary, trim them down. Dave Hansen said that the kernel has a hard time just managing the memory types it supports now; adding another is unlikely to make things better. The discussion circled around these points for a while but, as the session came to a close, it was clear that /dev/umap would have a hard time getting upstream in its current form.

Index entries for this article
Conference	Storage, Filesystem, and Memory-Management Summit/2018

---

to post comments

Exposing storage devices as memory

Posted Apr 27, 2018 23:44 UTC (Fri) by neilbrown (subscriber, #359) [Link] (1 responses)

> Manzanares responded that he didn't want to use the existing swap mechanism because it depends on the block layer

I don't think swap does depend on the block layer. Certainly swap-over-NFS never sees a bio. If you configure a swap file (rather than a swap blockdevice) it just calls the direct_IO() and readpage() address_space_operations.

Maybe if we put a filesystem interface on these new storage devices and gave swapon a file name in this filesystem, it might be able to perform better than putting a block device interface on them.

Exposing storage devices as memory

Posted May 3, 2018 20:50 UTC (Thu) by osandov (subscriber, #97963) [Link]

> If you configure a swap file (rather than a swap blockdevice) it just calls the direct_IO() and readpage() address_space_operations.

That's only the case for swap-over-NFS. Everywhere else, the swap code gets the location of the file on-disk via bmap() and then goes straight to the disk. The ->swap_activate() operation is only used by NFS, but XFS and Btrfs are also going to use it soon to make sure that swapping doesn't interfere with copy-on-write files. This will still bypass the filesystem, because going through the filesystem for swap means that your GFP_NOFS allocations from the filesystem could write to swap and recurse into the filesystem (and grab filesystem locks!).

Exposing storage devices as memory

Posted Apr 27, 2018 23:59 UTC (Fri) by ejr (subscriber, #51652) [Link]

Latency differences exist in NUMA already. A more unified view would be good, not a more fragmented one.