Who merges?

Posted Dec 17, 2008 9:49 UTC (Wed) by Nick (guest, #15060)
In reply to: Who merges? by ncm
Parent article: SLQB - and then there were four

I haven't talked to Paul. I didn't know that.. but I should, now you mention it!

Actually, within the same node, CPUs can allocate objects they've freed which have come from another CPU (on that node). When freeing an object to our freelist, the test that is performed is whether the object's memory is on the same node as the node this CPU is on. This is similar to what SLAB does, and indeed is good for cache (and object management overhead).

Except in very special cases of slabs (RCU-ish ones), the linked list is part of the object itself (when the object is freed, nobody else by definition is using the memory). _Often_ in the kernel I'd say that when an object is freed it is probably recently been touched by the freeing CPU. It would be an unusual situation eg. to have a refcount to the object residing somewhere other than the object itself.

And yes, we have the struct page structure for every page in the system, which can be found with calculations on the object address. And yes, all the slab allocators use this struct page to manage their slabs of objects :)

I agree with your last paragraph too...

to post comments

Who merges?

Posted Dec 18, 2008 1:00 UTC (Thu) by ncm (guest, #165) [Link] (2 responses)

Maybe it's a mistake to store the refcounts with the objects. Very often an object is allocated and initialized, and then never modified again, except for the refcount, and perhaps not even looked at again, until freed when the owning process dies. If the allocator provided centralized storage for refcounts, they could be yanked out of the objects, reducing cache churn. Centralizing refcounting could have other benefits, such as allowing optimizations for small counts and for small fluctuations in counts to be abstracted.

Integrating refcounting would change the SLAB interface, but generic refcounting services could be added to all the allocators, providing another way for them to differentiate themselves.

Who merges?

Posted Dec 18, 2008 8:41 UTC (Thu) by Nick (guest, #15060) [Link] (1 responses)

I don't know how common that really is. For things allocated with significant frequency, I think they should be fairly cache hot at free time (because if they are allocated frequently, they shouldn't have long lifespans).

The exception are caches, that are reclaimed when memory gets low or a watermark is reached (eg. inode cache, dentry cache, etc). However, with these things, they still need to be found in order to be reclaimed, usually via an LRU list, so the object gets hot when it's found and taken off the list.

OK, you could move the refcount and the lru list into another area... but the other problem with that is that in cache objects you expect to have multiple lookups over the lifetime of the object. And if your lookups have to increment a disjoint refcount object, then you increase your cacheline footprint per lookup by effectively an entire line per object. So you trade slower lookups for faster reclaim, which could easily be a bad tradeoff if the cache is effective (which the dcache is, 99% of the time).

Did you have any particular situations in mind?

Who merges?

Posted Dec 18, 2008 21:37 UTC (Thu) by ncm (guest, #165) [Link]

This is the danger of armchair coding; you're probably right.

An optimized refcount scheme may take only a few bits per object, for most objects, so I was thinking one cache line might hold refcounts for a hundred objects. Also, a dirty cache line is way more expensive than a clean one (because it must be written back, and isn't a candidate for replacement until that's done) so I meant to concentrate the refcount churn, and segregate it from the (commonly) mostly-unchanging objects. This helps most where you have some semblance of locality. Unfortunately things like inode caches don't.

As always, there's no substitute for measuring, but you have to build it first.