Store: limit cemented cycles size to 65_535
Cemented files are the place holders were cemented blocks (blocks not subject to reorgs) are stored, as chunks (usually conforming to the definition of protocol cycles). The metadata are store in the same way, be they are compressed to gain disk space. camlzip is used to compress the data allowing to gain a decent disk space and allowing good performances as access to the data can be performed in a random way, without uncompressing the whole file.
Due to the 32 bits limitations of camlzip, and the fact that the 64bits version is likely not to exist https://github.com/xavierleroy/camlzip/issues/35 such zip files cannot:
- handle more that 2^16 - 1 entries
- handle files bigger that 4Gib
Here, we address the first item to avoid cementing chunks bigger than the limit. To do so, we split in two even (if possible) parts, all cycles that are bigger that the limit, before cementing them.
Manually testing the MR
Activate Protocol P on the beginning of a cycle to trigger the cementing of a big chunk. Observe the store merge. Request blocks (and metadata) from this chunk to make sure they are well cemented.
Such a scenario will result in cementing a chunk of 5 cycles witch is 16_384*5=81_920 blocks. As this is greater than the limit, it must be split in two chunks of 40_960 blocks.
./tezos-node-mainnet-paris/store/chain_NetXdQprcVkpa/cemented
├── 5005314_5021697 ----> 16384
├── 5021698_5038081 …
├── 5038082_5054465 …
├── 5054466_5070849 ----> 16384
├── 5070850_5111809 ----> 40960
├── 5111810_5152769 ----> 40960
├── hash_index
│ └── index
│ ├── data
│ ├── lock
│ └── log
├── level_index
│ └── index
│ ├── data
│ ├── lock
│ └── log
└── metadata
├── 5005314_5021697.zip
├── 5021698_5038081.zip
├── 5038082_5054465.zip
├── 5054466_5070849.zip
├── 5070850_5111809.zip
└── 5111810_5152769.zip
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rstfor the protocol and the environment,CHANGES.rstat the root of the repository for everything else). -
Select suitable reviewers using the Reviewersfield below. -
Select as Assigneethe next person who should take action on that MR