Add opam transparent cache

What

This is a simple wip to evaluate the kisscache server for opam.

The kisscache server can be found here : https://gitlab.com/Linaro/kisscache

For evaluating this MR on the CI I installed ( docker compose up -d ) a kisscache server on http://p2p-monitoring-sandbox.nomadic-labs.eu where you can see the packages being downloaded and stored.

This MR is a refresh of !13817 (closed)

Why

Using a caching server has two main benefits. First reduce the amount of data downloaded outside the GCP network, reduce costs and network lags/flakiness.

Second to cache all tgz, so network failures won't affect the compilation. Also, since packages are cached, if they are modified upstream ( without notice, and without updating the version number ), these should not invalidate the build.

Kisscache handles everything transparently. New packages are download on demand and then cached. There are a number of settings to optimize the resources on the server: https://gitlab.com/Linaro/kisscache/-/blob/master/kiss_cache/settings.py?ref_type=heads

How

I add a simple wrapper for opam in scripts/kiss-fetch.sh that mangle the url to pass thought the cache. By default ( and for this poc ) we use a local server, but we can set a remote server using the env var KISSCACHE.

opam knows about this wrapper via the env var OPAMFETCH ( this does not work on the latest stable release of opam and you need to download the latest rc for opam 2.2 )

export OPAMFETCH="scripts/kiss-fetch.sh"

This is a POC only. The docker compose recipe must be adapted for production, adding a reverse proxy to handle basic authentication.

Manually testing the MR

To test this poc

• clone kisscache locally : git clone https://gitlab.com/linaro/kisscache
• docker compose up -d
• open "http://localhost:8001" to check the kisscache interface
• export OPAMFETCH="scripts/kiss-fetch.sh"
• rm -Rf _opam
• opam clean -c
• make build-deps

This will build octez as usual, but passing through the cache server. The second time you will perform this operation, no packages will be downloaded from the network, but instead they will be retrieved from the cache.

Notice that in terms of performance, on a fast network, you won't probably notice any major differences as the compilation time dominates the download time for make build-deps.

However you will gain in terms of stability. This caching system is not useful locally, as opam already cache all dependencies, but we can leverage this service on the CI where we often download fresh dependencies from the network.

to download a single binary with kisscache you can try :

curl 'http://p2p-monitoring-sandbox.nomadic-labs.eu/api/v1/fetch?url=https%3A%2F%2Fgithub.com%2Focaml%2Focaml%2Freleases%2Fdownload%2F5.2.1%2Focaml-5.2.1.tar.gz' -O

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR

Summary by CodeRabbit

• New Features
  • Enhanced package retrieval via an integrated caching mechanism that verifies server availability and offers a fallback option for smoother package operations.
  • Streamlined configuration settings ensure consistent performance across build processes, leading to a more reliable user experience when managing packages.
  • New script added to facilitate OPAM package management with kisscache integration.