Context and goal

The Data Availability Layer (DAL) can theoretically support a bandwidth of 0.5MiB/s on Mainnet by publishing the maximum of 32 DAL slots at each level.

The goal of this project is to:

• determine the actual currently supported bandwitdh
• identify bottlenecks and performance issues for increasing the bandwidth (other than shard verification)

Work breakdown

• (@eugenz) determine current Ghostnet bandwidth
• (@GuillaumeGen, @vect0r) measure bandwidth by simulating actual networks
  • enable snapshot/context loading (!18272 (merged))
  • enable yes-wallet and yes-crypto (in tezt-cloud DAL scenario) (!18445 (merged))
  • perform experiments
• (@gabriel.moise) measure bandwidth with crypto disabled
  • Document
  • flag to disable crypto in DAL node - !18206 (merged)
  • perform experiments to determine other bottlenecks besides crypto - logbook
  • Change shards storage for Attesters (on-disk -> memory) - !18442 (merged)
• measure effective DAL bandwidth (from rollup viewpoint)
  • (@picdc) E2E test with sequencer and consumer
  • (@picdc) write a tutorial to reproduce it
  • (@picdc) enable bandwidth measurement
    • adapt DAL echo kernel !18491 (merged)
    • enable supporting different kernels in the tezt-cloud scenario (!18492 (merged), !18577 (merged))
    • add observables (eg a metric) !18582 (merged)
• (@picdc) support open-telemetry
  • replace open-telemetry calls in !15821 (closed) with PPX profiler calls (!18346 (merged))
  • adapt tezt-cloud (!18416 (merged), !18417 (merged))
  • trace gossipsub messages publication (!18443)
• (@gabriel.moise) add more PPX profiling to the DAL node
  • --ppx-profiling option to DAL scenario in tezt-cloud - !18326 (merged)
  • Profiling in DAL node (and validate with tezt-cloud) - !18320 (merged)
  • Profiling in the GossipSub worker (and validate with tezt-cloud) - !18341 (merged)
  • Prometheus and OpenTelemetry profiling backends when --ppx-profiling is enabled - !18394 (merged)
  • Profiling in DAL node for shards_handler - !18403 (merged)
• (@eugenz) test reconstruction
  • enable the DAL node to not propagate shards (!18191 (merged))
  • add tezt test (!18461 (merged))
  • test on Ghostnet