Dac: observer downloads missing pages from committee
Context
We want missing pages to be fetched from Committee Members instead of the Coordinator.
This MR introduces a new kind of page store Remote_with_flooding where the remote fetch sends a request to each committee member in parallel. Additionally, GET /missing_page rpc is also removed from Coordinator as this endpoint should only be exposed by Observer nodes going forward.
Benchmark results
Benchmark below tells us that performance is related to number of committee members and size of message both which are expected. When fetching a single page (4kb), flooding can handle at least 50 committee members. As expected, flooding performs poorly when polling large payloads and should not be relied on to kick-off an Observer node from cold for large payloads. Created #5668 (closed) to track this issue.
**Single page fetch (Target use case)**
Benchmark the time it takes to fetch a single page.
| committee members | time (seconds) |
|---|---|
| 2 | 0.001536 |
| 5 | 0.002289 |
| 10 | 0.002946 |
| 20 | 0.003461 |
| 50 | 0.008432 |
| 100 | fails - [error] Too many open files |
Fetch all missing pages
Benchmark the time it takes to fetch a all pages from a root hash. This happens when the Observer node misses the published root hash (eg. starts cold)
| size (mb) | committee members | ~time (s) |
|---|---|---|
| 2 | 2 | 1.604237 |
| 2 | 3 | 1.846500 |
| 2 | 5 | 2.309726 |
| 2 | 10 | 3.203248 |
| 5 | 2 | 4.232832 |
| 5 | 3 | 4.547821 |
| 5 | 5 | 5.679550 |
| 5 | 10 | fail - [error] Can't assign requested address |
| 7 | 2 | 5.668716 |
| 7 | 3 | 6.690229 |
| 7 | 5 | fail - [error] Can't assign requested address |
| 10 | 2 | ~8.047370 |
| 10 | 3 | fail - [error] Can't assign requested address |
Part 2/2. Closes: #4522 (closed)
Manually testing the MR
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rstfor the protocol and the environment,CHANGES.rstat the root of the repository for everything else). -
Select suitable reviewers using the Reviewersfield below. -
Select as Assigneethe next person who should take action on that MR