The goal is to reduce the DAL’s latency by decreasing the value of the attestation_lag constant.

Team: @GuillaumeG, @iguerNL, @gabriel.moise, @vch9, @eugenz (owner)

Resources

• Project document: Reduced DAL latency
• Team updates: sync
• Implications of updating attestation_lag: document
• Journal of tezt-cloud experiments: document

Work breakdown

See also the list of MRs.

• Latency optimizations
  • (@gabriel.moise) propagate shards after 1 level (instead of 2) - #6899 (closed) !19366 (merged)
  • (@gabriel.moise) provide a stream RPC in the DAL node for monitoring shards - !19459 (merged)
  • (@gabriel.moise) adapt baker to use the stream RPC
    • Add No_shards_assigned and Shard_has_trap cases for stream elements - !19721 (merged)
    • Add Backfill case for stream elements - !19736 (merged)
    • Add support for Proxy and Mockup_for_baker modules in tezt - !19786 (merged)
    • Plug monitoring RPC in the baker
      • Fix Backfill logic in Attestable_slots_watcher_table - !19876 (merged)
      • Backfill mechanism performance optimisations - !19948 (merged)
      • Add DAL attestable slots worker in baker - !19875 (merged)
      • Update DAL attestable slots worker streams in baker - !19877 (merged)
      • Use monitoring RPC for forging DAL attestation - !19880 (merged)
    • Add profiling in the DAL attestable_slots worker - !19811 (merged)
    • Add tezt test for attestable_slots options - !20011 (merged)
• Tests
  • (@GuillaumeGen) write dedicated migration test - !19541 (merged)
  • (@GuillaumeGen) write test for entrapments around migration - !19547 (merged)
  • (@eugenz) extend skip-list migration test and get_slot_commitment RPC - !19571 (merged)
  • (@eugenz) adapt DAL tests that assumed a particular value of attestation_lag - !19704 (merged)
• support attestation_lag change in the DAL node
  • (@gabriel.moise) get_attestable_slots returns "unattestable" for the last levels in P1 - #4612 (closed) !19434 (merged)
  • (@eugenz) migrate the skip list store - !19570 (merged)
  • (@eugenz) update slot statuses correctly at migration - #8079 (closed) !19734 (merged)
  • (@GuillaumeGen) correctly clean-up the store - !19722 (merged)
  • (@sagotch) add Unpublished case to header_status type - !19613 (merged)
• support attestation_lag change in the economic protocol
  • (@iguerNL) protocol stitching code - !19466 (merged) (see also !19535 (closed))
  • (@eugenz) ignore DAL content for the first lag' levels of T - !19442 (merged) (in particular this commit)
  • (@GuillaumeGen) deal with entrapment accusation at protocol border - !19548 (merged), !19823 (merged)
  • DAL/Rollup integration
    • (@iguerNL) deal with slot id validation for DAL page reveals
      • prepare skip-list cells to carry attestation lag - !19500 (merged)
      • introduce & plug a dynamic case for attestation_lag_kind - !19509 (merged)
      • adapt refutation games for lag reduction - !19513 (merged)
      • cells_of_level returns the attestation_lag as well - !19512 (merged)
      • rollup node: remove useless calls to compute an already given published level - !19703 (merged)
      • get_slot_status RPC returns attestation lag - !19681 (merged)
      • fetch DAL attestation status and lag from DAL nodes - !19532 (merged)
      • simplify DAL pages request interface - !19917 (merged)
      • easy refutations tests framweork for DAL pages reveal - !19776 (merged)
      • refutation tests that cover various corner cases - !19779 (merged)
      • wait a few seconds when fetching a non-final DAL slot status - !19919 (merged)
      • fix refutation games - !19977 (merged)
      • refutation tests that cover various corner cases - !19799 (merged)
      • harden the case where published level is in the future wrt import level - !20051 (merged)
      • add refutation test over a protocol migration - #8156 (postponed)
    • (@vch9) use right DAL parameters for DAL parameters' reveals - !19533 (merged)
      • store DAL constants for each protocol - !19674 (merged)
      • new test where a refutation game is played on reveal DAL parameters - !19659 (merged)
• (@GuillaumeGen) (re-)run experiments (adding network latency) to confirm the new value
  • (@gabriel.moise) add attestation_lag option in DAL tezt-cloud framework - !19344 (merged)
  • (@GuillaumeGen) allow slot publication from the DAL node - !19975 (merged)
  • (@GuillaumeGen) Tezt_cloud: publish slots using the DAL node instead of relying on the orchestrator - !19976 (merged)
• (@eugenz) change lag value - !19442 (merged)

Optional, nice to haves:

• DAL node
  • (@eugenz) backfill statuses at startup - !19969 (merged)
  • use voting RPC to identify the migration level - #8069
  • do not publish for lag levels before migration - #8157
  • (minor) maybe update traps_cache_size (or at least the doc-string) - !19882 (merged)
  • (minor, optional) de-correlate may_update_topics from attestation_lag - #8087 (closed), !19882 (merged)