Storage Protocol: optimize Big_map updating using local tree

Context

Improved the speed of Big_map update in protocol storage by using local access

This MR depends on !5922 (merged).

Remark from Jun Furuse: this is an optimization of lazy storage update using local trees. When a lazy storage map is updated, the modified key-value pairs are applied in batch. This application was done from the root directory /. This MR changes it from the base directory of the lazy storage map: for example, /big_maps/index/<ID>. This saves 3 directory accesses per key update of lazy storage: currently big_maps and saplings.

Idea of the optimization

The bigmap storage are updated collectively after evaluations. The accesses of the updating are using absolute path from /.

update /data/big_maps/<big_map_id>/contents/<key1>/data
update /data/big_maps/<big_map_id>/contents/<key1>/len
update /data/big_maps/<big_map_id>/contents/<key2>/data
update /data/big_maps/<big_map_id>/contents/<key2>/len

This MR makes the updating using relative paths from the map directory:

update <key1>/data
update <key1>/len
update <key2>/data
update <key2>/len

What this MR does

Local context

A new thpe Raw_context.local_context is introduced for the local access using the context tree. Like Raw_context.t, local_context has the gas fields to account the storage access cost using it.

Function with_local_context

Raw_context.with_local_context provides a wrapping function to use the local context. It takes a global context, the path for the tree to access locally, and a function to perform local access using the tree. It has a boolean option whether the final local tree is to be added back to the global context or not. If false is set, any modifications to the local tree are discarded. Whether the final tree is added back or not, the consumed gas in the local accesses is charged to the gas counter of the global context.

module Local

This MR introduces API functions for the relative access to storage modules created by Make_indexed_carbonated_data_storage functor.

Example: optimizations of Lazy_storage_diff.apply_updates

As an example, Lazy_storage_diff.apply_updates function is rewritten using these local access functions.

Benchmark

The overhead is to retrieve and update the tree for relative access before and after each successive update. We can measure the speed using the following branch: https://gitlab.com/proof_ninja/tezos/-/tree/yoshihiro503@bigmap-local-ithaca. This benchmark will run 500000 updates of 100 items. The data to be updated will be keys of type nat and values of type string.

dune exec src/proto_alpha/lib_protocol/test/unit/main.exe -- test bench_bigmap
cat _build/_tests/protocol\ U+003E\ unit/*

[new]: 713.363060 [sec]
[previous]: 1173.941913 [sec]

In my environment, the execution time was reduced to about 60%.

Gas

The amount of gas consumed does not depend on the length of the path, so the cost of gas is not reduced. Gas consumption rather increases due to the overhead of handling local context.

Manually testing the MR

Using yoshihiro503@bigmap-local-experimental-granada branch

dune exec src/proto_alpha/lib_protocol/test/update_bigmap.exe -- DATA_DIR

New checklist

• Benchmark: how it really improves the performance
  • A big_map update batch
  • The entire tezos node
• Intermediate code review (ignoring the gas issue)
• Gas issue
  • Detail the gas consumption issue
  • Discuss other teams to find a solution
• Final code review

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, the Development Version section of CHANGES.md for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR