EVM Node: Support transparent tracing of workers with OpenTelemetry
Part of L2-2
What
This commit adds a wrapper around Tezos_workers.Worker.MakeSingle in
Octez_telemetry. This wrapper enables automatic OpenTelemetry tracing for
worker requests, capturing details like request initiation, completion, and
errors. It also adapts various EVM node workers to utilize this new tracing
mechanism.
Why
The primary motivation for this change is to significantly improve the observability of Octez worker operations. By integrating OpenTelemetry, developers and operators can gain deeper insights into the execution flow, identify performance bottlenecks, and facilitate debugging of asynchronous worker tasks within the EVM node. This enhances the ability to monitor and understand the system's behavior in production.
The introduction of a dedicated library instead of contributing to
Tezos_workers is motivated by the lack of perspective we have on the cost of
instrumentation. Besides and contrary to the current approach used by the Octez
nodes, the EVM node releases will have the OpenTelemetry instrumentation. That
is, it will be possible to opt-in exporting traces in production. In order to
not affect the Tezos L1 binaries while we conduct the necessary experiments to
validate this choice, it is reasonable not to modify the shared codebase.
How
A new Worker module is added to Octez_telemetry. This module exports a
MakeSingle functor that takes a Request module and creates an instrumented
worker. The new worker transparently creates OpenTelemetry spans for each
request, attaching relevant attributes. The Request modules of existing EVM
node workers (e.g., block producer, blueprint publisher, tx queue) are extended
with a name function to provide descriptive span names. The build system
configurations (dune, manifest, opam) are updated to include the new library
and its dependencies.
Manually testing the MR
Checklist
-
Document the interface of any function added or modified (see the coding guidelines) -
Document any change to the user interface, including configuration parameters (see node configuration) -
Provide automatic testing (see the testing guide). -
For new features and bug fixes, add an item in the appropriate changelog ( docs/protocols/alpha.rstfor the protocol and the environment,CHANGES.rstat the root of the repository for everything else). -
Select suitable reviewers using the Reviewersfield below. -
Select as Assigneethe next person who should take action on that MR