Profiler: add an Opentelemetry "backend"

Based on top of !15893 (merged), so that OpenTelemetry instrumentation happens only if the backend is selected.

What

This MR adds a simple and mocked backend for OpenTelemetry, that only does some traces without much information.

Why

OpenTelemetry has an advantage for monitoring: it is designed to produce traces that are consistent across multiple binaries as long as they can produce consistent ids. The purpose for this integration is to first visualize the propagation of operations across multiple nodes.

How

OpenTelemetry unfortunately is not really compatible with OpenTelemetry: it has a similar interface but it cannot be made into a driver without retro engineering the implementation of its API. As such, we only use the ppx profiler as some sort of ifdef macro, that adds the correct code to start tracing that is plugged in only when the PPX is enabled. As a result, it mainly uses [@profiler.custom_f Opentelemetry_profiler.trace] to identify segments to profile. Opentelemetry_profiler is then a glorified API around Opentelemtry.Trace that would add specific metadata to traces.

Manually testing the MR

Testing the MR can be quite simple:

• Compile the node with the profiler and the backend enabled:

TEZOS_PPX_PROFILER=opentelemetry make octez-node

• Run a node in sandbox:

./src/bin_node/octez-sandboxed-node.sh 1 --connections 0

and in another terminal

eval `./src/bin_client/octez-init-sandboxed-client.sh 1`
octez-activate-alpha

• Now, you need Jaeger. This is basically the program that will receive the events from the OpenTelemetry collector and handle the visualization. The magic command is actually simple:

docker run --rm --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.63.0

Basically, the OpenTelemetry collector will push its events by default to localhost:4318 (or 4317, but ocaml-opentelemetry doesn't support gRPC as far as I know).

Once everything is running, you should be able to access to http://localhost:16686 from your favorite browser

Then, make a transfer from the client:

octez-client -w none transfer 422 from bootstrap3 to bootstrap4

From Jaeger's interface, you should now see a service called octez-node

and once you click on Find traces, you should see at least a trace:

Click once again, try to click on some of the "spans" and you should get something like this:

You now have a basic trace of the operation from the injection RPC up to the end of the prevalidator. You can see that there are two "blocks" in the same trace: the first one is the injection RPC up to the moment it sends the request to the worker, the second starts at the moment the prevalidator worker handles the request, with some events (when the operation has been parsed, when it has been validated). Basically, some information is lost when going to the worker, as they are not running in the same thread and the paternity between spans is lost. However thanks to the timestamps, we can see how it moves forward.

What's next

Basically, this trace shows that something happens, but it is hard to see exactly what happens. If you run a second node, you'll see that it will appear in the trace but it will be difficult to distinguish which node does what. The next steps are basically to add more info, improve the consistency of traces, trace down to the P2P level, make traces across multiple node usable.

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR