[go: up one dir, main page]

Fix irmin unknown key

A node on Limanet failed with:

Jan  8 10:55:40.603 - prevalidator: switching to new head BMF1fSJ8VTBBgbZ2rbdLxTEFPoq1taHsaTfPfJvUAF3KwtpHpD1
Jan  8 10:55:40.603 - prevalidator:   Request pushed on 2023-01-08T09:55:40.530-00:00, treated in 1.93ms, completed in 71.508ms
Jan  8 10:55:40.604 - validator.chain: Update current head to BMF1fSJ8VTBBgbZ2rbdLxTEFPoq1taHsaTfPfJvUAF3KwtpHpD1
Jan  8 10:55:40.604 - validator.chain:   (level 473336, timestamp 2023-01-08T09:55:40-00:00, fitness
Jan  8 10:55:40.604 - validator.chain:   02::000738f8::::ffffffff::00000000), same branch
+333951088443us index [INFO] [context] sync
+333951103518us index [INFO] [context] sync
+333951196932us index [INFO] [context] sync
+333951205049us index [INFO] [context] sync
+333951207823us index [INFO] [context] sync
+333951211448us index [INFO] [context] sync
Jan  8 10:55:40.746 - prevalidator: injecting operation ontQkycyHRLdjoDULoPsdJwX1FzvzitJy5XYZjw4xNVxfziZCzp (force:false)
Jan  8 10:55:40.746 - prevalidator:   Request pushed on 2023-01-08T09:55:40.744-00:00, treated in 35.751us, completed in 1.410ms
Jan  8 10:55:41.328 - prevalidator: injecting operation onmpBDi1pYJKFdMrSdPXfSVzn8yH2ph7BUBS4ZJFvhw2CYNZEZq (force:false)
Jan  8 10:55:41.328 - prevalidator:   Request pushed on 2023-01-08T09:55:41.326-00:00, treated in 74.177us, completed in 998us
Jan  8 10:55:55.872 - validator.block: prechecked block BLS4cb5Li1Thsw1NNPmNhXm4A4i4bW7YmtJDcyU8jtn9pf6PLu8
Error:
  {"Direct":["CoW7PhFfoc1FC9RCiFEbqDZjNA3XMnwFE5pCSpUvwQGWXU3nj8df",72051238680,319]}: unknown key

Jan  8 10:56:10.642 - node.validator.bootstrap_pipeline: fetching branch of about 2 blocks from peer idstZmykY7p7NBT8D55vdxwHspHGRD in
Jan  8 10:56:10.642 - node.validator.bootstrap_pipeline:   1 steps
Jan  8 10:56:10.645 - node.validator.bootstrap_pipeline: unexpected error while fetching headers:
Jan  8 10:56:10.645 - node.validator.bootstrap_pipeline:   {"Direct":["CoW7PhFfoc1FC9RCiFEbqDZjNA3XMnwFE5pCSpUvwQGWXU3nj8df",72051238680,319]}: unknown key
Jan  8 10:56:10.647 - node.validator.bootstrap_pipeline: fetching branch of about 3 blocks from peer idtHBCb2FoMxViQ7hdutfTwcPAukyP in
Jan  8 10:56:10.647 - node.validator.bootstrap_pipeline:   2 steps
Jan  8 10:56:10.647 - validator.peer: request
Jan  8 10:56:10.647 - validator.peer:   New branch BLS4cb5Li1Thsw1NNPmNhXm4A4i4bW7YmtJDcyU8jtn9pf6PLu8, locator length 19951
Jan  8 10:56:10.647 - validator.peer:   failed
Jan  8 10:56:10.647 - validator.peer:   (Request pushed on 2023-01-08T09:56:10.640-00:00, treated in 171us, completed in 5.623ms):
Jan  8 10:56:10.647 - validator.peer:   Error:
Jan  8 10:56:10.648 - validator.peer:     {"Direct":["CoW7PhFfoc1FC9RCiFEbqDZjNA3XMnwFE5pCSpUvwQGWXU3nj8df",72051238680,319]}: unknown key
Jan  8 10:56:10.648 - validator.peer:
Jan  8 10:56:10.648 - validator.peer: worker crashed [validator-peer]:
Jan  8 10:56:10.648 - validator.peer:   Error:
Jan  8 10:56:10.648 - validator.peer:     {"Direct":["CoW7PhFfoc1FC9RCiFEbqDZjNA3XMnwFE5pCSpUvwQGWXU3nj8df",72051238680,319]}: unknown key
Jan  8 10:56:10.648 - validator.peer:
Jan  8 10:56:10.648 - node.validator.bootstrap_pipeline: unexpected error while fetching headers:
Jan  8 10:56:10.648 - node.validator.bootstrap_pipeline:   {"Direct":["CoW7PhFfoc1FC9RCiFEbqDZjNA3XMnwFE5pCSpUvwQGWXU3nj8df",72051238680,319]}: unknown key

This seems to be related to the fact that some values that were GCed are still stored in the LRUs, and thus, something tries to access to it. The hypothesis is that a RO instance (the node or the baker) triggered this error as only the RW instance cleans the LRUs caches after a GC. We should clean the LRUs of RO instances as well.