Per-file OOM badness

Posted Jun 2, 2022 18:40 UTC (Thu) by NYKevin (subscriber, #129325)
Parent article: Per-file OOM badness

> As a simple example, he said in the patch-series cover letter, a malicious process can call memfd_create(), then just write indefinitely to the resulting memfd; the memory consumed by the memfd will not be seen as belonging to the offending process so, when the memfd ends up consuming all of the available memory, the OOM killer will pass over that process. This sequence "can bring down any standard desktop system within seconds". Another problem area, he said, is graphics applications that allocate significant amounts of memory within the kernel for graphical resources.

That does not sound like a fixable problem in the general case. Can't the malicious process just create files in /dev/shm (either directly, or via shm_open(3)) instead? I find it hard to believe that the kernel can keep track of who created those files, and OOM killing the process won't even clean them up anyway.

to post comments

Per-file OOM badness

Posted Jun 2, 2022 20:00 UTC (Thu) by nybble41 (subscriber, #55106) [Link]

> Can't the malicious process just create files in /dev/shm (either directly, or via shm_open(3)) instead?

Files created in /dev/shm/ reside in a named tmpfs filesystem, which sets an upper bound on the memory consumed (50% of RAM by default—Debian doesn't appear to override this). The files created with memfd_create() are also tmpfs files, from an internal, unmounted tmpfs, but they behave more like anonymous shared memory mappings (mmap() with MAP_ANONYMOUS)—so far as I can tell the limits on /dev/shm/ do not apply. There is a limit on the total amount of shared memory, the kernel.shmall sysctl knob, which I think would also affect memfd_create(), but this defaults to "unlimited" (~2**64 pages).