[go: up one dir, main page]

Store: limit cemented cycles size to 65_535

Cemented files are the place holders were cemented blocks (blocks not subject to reorgs) are stored, as chunks (usually conforming to the definition of protocol cycles). The metadata are store in the same way, be they are compressed to gain disk space. camlzip is used to compress the data allowing to gain a decent disk space and allowing good performances as access to the data can be performed in a random way, without uncompressing the whole file.

Due to the 32 bits limitations of camlzip, and the fact that the 64bits version is likely not to exist https://github.com/xavierleroy/camlzip/issues/35 such zip files cannot:

  • handle more that 2^16 - 1 entries
  • handle files bigger that 4Gib

Here, we address the first item to avoid cementing chunks bigger than the limit. To do so, we split in two even (if possible) parts, all cycles that are bigger that the limit, before cementing them.

On top of !12262 (merged)

Manually testing the MR

Activate Protocol P on the beginning of a cycle to trigger the cementing of a big chunk. Observe the store merge. Request blocks (and metadata) from this chunk to make sure they are well cemented.

Such a scenario will result in cementing a chunk of 5 cycles witch is 16_384*5=81_920 blocks. As this is greater than the limit, it must be split in two chunks of 40_960 blocks.

./tezos-node-mainnet-paris/store/chain_NetXdQprcVkpa/cemented
├── 5005314_5021697 ----> 16384 
├── 5021698_5038081 …
├── 5038082_5054465 …
├── 5054466_5070849 ----> 16384 
├── 5070850_5111809 ----> 40960
├── 5111810_5152769 ----> 40960
├── hash_index
│   └── index
│       ├── data
│       ├── lock
│       └── log
├── level_index
│   └── index
│       ├── data
│       ├── lock
│       └── log
└── metadata
    ├── 5005314_5021697.zip
    ├── 5021698_5038081.zip
    ├── 5038082_5054465.zip
    ├── 5054466_5070849.zip
    ├── 5070850_5111809.zip
    └── 5111810_5152769.zip

Checklist

  • Document the interface of any function added or modified (see the coding guidelines)
  • Document any change to the user interface, including configuration parameters (see node configuration)
  • Provide automatic testing (see the testing guide).
  • For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Victor Allombert

Merge request reports

Loading