(Q3-2024) Layer 1 - Improve store scalability

Improve store scalability

Cemented files are the place where cemented blocks (blocks not subject to reorgs) are stored, as chunks (usually conforming to the definition of protocol cycles). These files are basically concatenated blocks (Block_repr.t) that we access thanks to file offsets and indexes. These offset are encoded as unsigned 32 bits integers, thus, only 4Gib files can be addressed. It is mandatory to be able to address bigger files to avoid critical and unrecoverable failure as soon as we will reach this limit.

The metadata are store in the same way, but they are compressed to gain disk space. camlzip is used to compress the data allowing to gain a decent disk space and allow good performance. Thanks to zip specifications, accessing to the data can be performed in a random way, without uncompressing the whole file. Currently, the default metadata limits (set by the shell) allow them to grow up to 240Gb for 24,576 blocks (1 cycle).

In relation to the Store, it is known that merges of store are consuming a lot of resources (doubling the memory usage). Currently, merging occurs simultaneously for each node in the network at each new cycle. This can slow down nodes, in the worst case can be a source of crashes and can slow down the overall network. Besides, beginning of cycles are also periods where bakers pay their rewards and may query their nodes with heavy RPCs. Moreover, having all nodes merging their stores at the same time make this high resource consumption predictable which can be used by attackers. Considering all this, we intend to make merging of store unpredictable by having each node separately and randomly selecting merge starting point within a cycle (making sure to avoid the early blocks of the cycle).

Work breakdown

Note: ETAs are estimates based on a project starting on the beginning of Q3 that is 2024/07/01. As the projects is likely to start on 2024/07/15, ETA are expected to be shifted accordingly to the start date.