Manifest: replace ~release with ~release_status

Context

In the manifest, we can declare that an executable is released, with ~release:true. But there are other sources of truth: the Makefile and the Dockerfile. Moreover, some executables are experimental: they should only be "released" for master, not for actual version tags. Finally, we only want to release opam packages that are needed to build released executables. All of this means that some cleaning up is needed.

This merge request is the first step towards this cleanup. It replaces ~release:true with ~release_status:Released, leaving room for ~release_status:Experimental. One can additionally annotate with ~release_status:Unreleased to explicitly state that a package must not be released. The manifest will emit an error if a release package depends on an unreleased package. Finally, with ~release_status:Auto_opam (which is the default), the package is released if, and only if, it is the dependency of another released package.

This merge request introduces function compute_opam_release_graph which computes the opam dependency graph. Each node contains the release status of the corresponding opam package. This function checks:

• that there is no opam package that was requested to be both released and not released;
• that there is no released opam package that depends on an unreleased opam package.

The resulting graph is used to simplify generate_opam_ci. Later, the plan is to also use the generated graph to restrict the set of released .opam files to the set of packages that were annotated with ~release_status:Released and their transitive dependencies. And the plan is also to generate octez.opam from the set of packages that were explicitly annotated with ~release_status:Released. See !6803 (merged) for a first attempt at this.

About opam_package.yml

This file differs from the original because of the rewrite of function generate_opam_ci. Care should be taken to check that:

• packages that were removed from this list are packages that we do not want to release;
• packages that were added (if any) are packages that we want to release;
• the longest jobs still start sooner than the slowest jobs, i.e. the total pipeline time does not increase.

About binaries-for-release

This file differs from the original because we do not actually want to release executables for protocol Alpha. We do want to publish them in the CI for master, but note that binaries-for-release is only used to generate the release at https://gitlab.com/tezos/tezos/-/releases, which is not something we do for master.

Future Work

In the future, my plan is to build upon this MR as follows:

• rename binaries-for-release into released-executables
• introduce experimental-executables next to released-executables
• use released-executables to decide which executables to build with make
• rename make build-unreleased to make experimental (keeping the old name as deprecated during a transition period)
• use experimental-executables to decide which executables to build with make experimental
• either split static-packages into experimental-static-packages + released-static-packages, or try to use *-executables directly
• use that to only publish experimental static executables for master, not for release tags
• try to use *-executables in the Dockerfile, or generate the Dockerfile, or something, to have only one source of truth, and to only have experimental executables in Docker images for master, not release tags
• redo !6803 (merged) on top of this

Manually testing the MR

Try to add and remove ~release_status annotations in packages to see the errors that the manifest gives. For instance, if you add ~release_status:Unreleased to lib_base, you get:

Package "tezos-baking-014-PtKathma" is released
  because its reverse dependency "tezos-baking-014-PtKathma-commands" is released
  because its reverse dependency "octez-accuser-PtKathma" is released
  because in src/proto_014_PtKathma/bin_accuser, "octez-accuser-PtKathma" is explicitly released
Package "tezos-base" is not released
  because in src/lib_base, "tezos-base" is explicitly not released
Error: Released package "tezos-baking-014-PtKathma" cannot depend on unreleased package "tezos-base".

and if you add ~release_status:Unreleased to tezos-store.shared and ~release_status:Released to tezos-store.unix, you get:

Error: In src/lib_store/shared, "tezos-store.shared" has release status Unreleased, and in src/lib_store/unix, "tezos-store.unix" has release status Released; those two targets cannot be in the same opam package "tezos-store".

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Provide automatic testing (see the testing guide).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR