Non-blocking RPC feature as experimental

This is a sub-milestone of %(2024Q2) - Layer 1 - Public RPC endpoint supporting average 1k RPS

This milestone aims to finish the non-blocking RPC feature that is stalled since a while -- see %Non-Blocking RPCs for more details. To do so, we first want to deliver as an experimental feature that will suffer from the resource leak discovered in %Non-Blocking RPCs.

Here is a wide picture of how the RPC-process behaves -- more details in %Non-Blocking RPCs and associated documents.

Work breakdown

• (ETA: 2024-04-12) Inventory of the remaining tasks @vect0r
  • rebase main branch !9490 (merged)
  • revive stalled MRs ( !11310 (merged), !11379 (merged), !11386)
  • continue splitting !9490 (merged) into sub MRs
• (ETA: 2024-05-03) Merge the last bits of the non-blocking RPC feature @vect0r
  •   !11310 (merged)
  •   !11379 (merged)
  • !11386 -> !13160 (merged) (the original one got stuck at merging)
    • !13057 (merged) - clean-up MR
  •   !12829 (merged)
  •   !12830 (merged)
  •   !12869 (merged)
  •   !12974 (merged)
  •   !12975 (merged)
  •   !12998 (closed)
  •   !13041 (merged)
  •   !13042 (merged)
  •   !13043 (merged)
  •   !13373 (merged)
  • Merge the non-blocking RPC feature !9490 (merged)
• (ETA: 2024-05-03) Notice the feature as experimental @vect0r
  • Notice the non-blocking RPC feature as experimental
• Benchmarks the RPC performances @vect0r
  • Compare the baseline ( --rpc-addr ) with --external-rpc-addr
    • Results: TL;DR: rpc-process is 10% faster but itroduces ~300ms latency for forwarded RPCs on the reference hardware (GCP n2-standard-4)
  • Persist the benchmarking framework in devtools
  • Persist the RPS values in result file handlers and notice it in the documentation !13606 (merged)
• Create a scheduled pipeline to test the non-blocking RPC feature
  • create a scheduled pipeline !13913 (merged)
  • configure it to be run every day
  • (optional) automatically post results on a slack channel done by default.