Draft: POC: kisscache opam

What

This is a simple wip to evaluate the kisscache server for opam.

to test this feature, you need to most recent opam rc version of opam 2.2 ( see : https://discuss.ocaml.org/t/how-to-use-opamfetch/14804 ).

The kisscache server can be found here : https://gitlab.com/Linaro/kisscache

Why

Using a caching server has two main benefits. First reduce the amount of data downloaded outside the GCP network, reduce costs and network lags/flakiness.

Second to cache all tgz, so network failures won't affect the compilation. Also, since packages are cached, if they are modified upstream ( without notice, and without updating the version number ), these should not invalidate the build.

This is a typical example of a job that failed because of a network problem : https://gitlab.com/tezos/tezos/-/jobs/7125765378

or this one : https://gitlab.com/tezos/tezos/-/jobs/7153986903 related to rust

How

I add a simple wrapper for opam in scripts/kiss-fetch.sh that mangle the url to pass thought the cache. By default ( and for this poc ) we use a local server, but we can set a remote server using the env var KISSCACHE.

opam knows about this wrapper via the env var OPAMFETCH ( this does not work on the latest stable release of opam and you need to download the latest rc for opam 2.2 )

export OPAMFETCH="scripts/kiss-fetch.sh"

This is a POC only. The docker compose recipe must be adapted for production, adding a reverse proxy to handle basic authentication.

Manually testing the MR

To test this poc

• install the latest opam RC : https://github.com/ocaml/opam/releases
• clone kisscache locally : git clone https://gitlab.com/linaro/kisscache
• docker compose up
• open "http://localhost:8001" to check the kisscache interface
• export OPAMFETCH="scripts/kiss-fetch.sh"
• rm -Rf _opam
• opam clean -c
• make build-deps

This will build octez as usual, but passing through the cache server. The second time you will perform this operation, no packages will be downloaded from the network, but instead they will be retrieved from the cache.

Notice that in terms of performance, on a fast network, you won't probably notice any major differences as the compilation time dominates the download time for make build-deps.

However you will gain in terms of stability. This caching system is not useful locally, as opam already cache all dependencies, but we can leverage this service on the CI where we often download fresh dependencies from the network.

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR