Rollup node: Store migration to SQLite store
What
This MR makes the rollup node migrate store data automatically (on startup) to the new SQLite database introduced by !15026 (merged).
Why
This is required to transition from the old file based store to the new SQLite format.
How
The migration logic follows the same pattern as the one from the old store versions:
- We iterate over all L2 blocks of the chain, from the head backwards.
- For each block we migrate all associated data : messages, inbox, commitment, level, DAL slot data.
- We finally migrate other singleton stores (outbox messages and protocols)
- We end up by migrating information about the rollup node state (L2 head, LCC, LPC, GC levels, history_mode, etc.)
Migration stats for various Etherlink configs
| Network & Config | Migration time | Blocks | Old store size | New store size |
|---|---|---|---|---|
| Ghostnet full | 4m39s | 275370 | 1.4GB | 1.6GB |
| Ghostnet archive | 74min | 2998614 | 14GB | 14GB |
| Mainnet archive | 32min | 1158590 | 2.7GB | 3.7GB |
Manually testing the MR
This MR can be tested using an existing rollup node data dir.
| Warning: Please back up the data dir before attempting the migration. |
|---|
It should display a progress bar in the logs to follow the progression of the migration.
Starting store migration
Sep 24 15:27:27.654: Store is being initialized for the first time
Sep 24 15:27:27.655: Applied migration initial to the store
Migrating store from v4 to v5_sqlite 275371/275370 04:39 │███████████████████████████████████████████████│ 100%
Store migration completed
To check for integrity of the new store, a simple way is to export a snapshot:
octez-smart-rollup-node -E https://rpc.tzkt.io/ghostnet snapshot export test.snapshot --data-dir <path> --uncompressed
Acquiring GC lock
(specify --rollup-node-endpoint for the snapshot command to take priority)
Acquiring process lock
Exporting snapshot 73.2 GiB 00:45 │█████████████████████████████████████████████████████████████████│ 100%
Extracting snapshot ✔️ 00:53
Checking snapshot 275493/275493 01:12 │█████████████████████████████████████████████████████████████████│ 100%
Edited by Alain Mebsout