Expedited memory reclaim from killed processes

Posted Apr 15, 2019 15:54 UTC (Mon) by Cyberax (✭ supporter ✭, #52523)
In reply to: Expedited memory reclaim from killed processes by rweikusat2
Parent article: Expedited memory reclaim from killed processes

That's a bad design. You shouldn't design software that depends on implicit assumptions.

A better designed software would store rules in a file and map it explicitly into the target processes. This way there's no problem with overcommit - the kernel would know that the data is meant to be immutable.

to post comments

Expedited memory reclaim from killed processes

Posted Apr 15, 2019 17:03 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link] (3 responses)

> That's a bad design. You shouldn't design software that depends on implicit assumptions.

This statement means nothing (as it stands).

> A better designed software would store rules in a file and map it explicitly into the target processes. This way there's no problem
> with overcommit - the kernel would know that the data is meant to be immutable.

A much more invasive change to suricata (this is an open source project I'm not anyhow associated with) could have gotten rid of all the pointers in its internal data structures. Assuming this had been done and the code had also been changed to use a custom memory allocator instead of the libc one, one could have used a shared memory segment/ memory mapped file to implement the same kind of sharing. I'm perfectly aware of this. But this complication isn't really necessary with Linux as sharing-via-fork works just as well and is a lot easier to implement.

Expedited memory reclaim from killed processes

Posted Apr 15, 2019 17:10 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link]

For completeness¸ before someone pulls that out of his hat and triumphantly waves it through the air: Changing to a custom allocator had been sufficient as the parent could have put all of its data into a shared memory segment/ memory mapped file and children could have inherited the mapping via fork.

But that's still more complicated than just relying on the default behaviour based on knowing how the application will use the inherited memory.

Expedited memory reclaim from killed processes

Posted Apr 15, 2019 17:30 UTC (Mon) by farnz (subscriber, #17727) [Link] (1 responses)

You could also, assuming it's backed by an mmaped file, just use MAP_FIXED to ensure that all the pointers match in every Suricata process; this works out best on 64-bit systems, as you need a big block of VA space available that ASLR et al won't claim.

Expedited memory reclaim from killed processes

Posted Apr 15, 2019 19:14 UTC (Mon) by rweikusat2 (subscriber, #117920) [Link]

Considering that the amount of memory which will be needed isn't known in advance, one would need to use MAP_FIXED in a "known good" location, anyway. There are some more complication with this approach as well. And the Linux default policy of "real COW", ie do not copy anyting unless it has to be done and don't (try to) reserve anything unless it's demonstrably required handles this cause just fine.