WIP: lib_context: add Context.gc
This MR adds a new Context.gc function to lib_context to delete old commits. The goal is to call this regularly (for instance every cycle) to keep a constant space usage for rolling nodes.
This required irmin and irmin-pack pinned to https://github.com/mirage/irmin/tree/3.4 (it is still a moving branch until we are happy with this MR) to work properly.
This is not ready for production use yet, but we are opening the MR to get early feedback.
Current status:
- most of the storage logic is already in Tezos' master since irmin 3.3.1 has been merged in !5679 (merged)
- Irmin 3.4 is adding on top of that an asynchronous GC. We manage to replay all operations of Hangzhou while doing the GC without issue. We run various stress tests and we've fixed all the issues we found (so we are not aware of any bugs right now - please help us to find more :p)
- the intended use-case is to start with a snapshot import and call GC regularly. Existing stores will continue as before (without GC). We do not intend to support the upgrade of existing stores (let us know if this is an issue)
- the asynchronous GC spawns a separate process. Right now this is not configurable, but it's easy to replace this with something else (for instance to avoid forking when the node is is
--single-processmode). Let us know what kind of integration you prefer: it's easy to integrate with any existing mechanism of process pooling. - (blocker) we found a performance regression during the bootstrap - we have identified the likely cause and will come up soon. This shouldn't have any impact in normal mode operation and so should not be critical for testing.
Edited by Thomas Gazagnaire