ciao: refactor dependecy/needs for optional artifacts

What

This MR introduces a new dependency type Optional_Artifacts to the Tezos CI system to better handle GitLab CI optional dependencies that don't require artifact downloads.

The change addresses a limitation where GitLab CI's optional: true in needs allows optional dependencies, but since we set dependencies: [] by default, artifacts are never downloaded. To enable optional artifacts, jobs must be listed in both needs and dependencies. Check the gitlab documentation about the field needs:artifacts : https://docs.gitlab.com/ci/yaml/#needs

Key changes:

• New Optional_Artifacts dependency type in tezos_ci.mli
• Extended need record with an artifacts: bool field in types.ml
• Updated dependency resolution logic in tezos_ci.ml
• Enhanced YAML generation to handle the new artifacts field
• Converted two build jobs (job_build_x86_64_release and job_build_x86_64_exp_dev_extra) from Optional to Optional_Artifacts. Notice that for this job, the dependency field is not [], so in this case, since the job is not listed among the declared dependencies adding artifacts:false is a noop here.

This creates two distinct types of optional dependencies:

• Optional: Downloads artifacts by default if the job runs
• Optional_Artifacts: Does not download artifacts, used purely for job ordering

Why

The current GitLab CI dependency system has a gap when dealing with optional dependencies:

1. Job ordering vs artifact needs: Sometimes we need to depend on a job for ordering purposes but don't need its artifacts
2. Performance optimization: Downloading unnecessary artifacts impacts pipeline performance
3. Clarity of intent: The distinction between ordering dependencies and artifact dependencies improves code readability

The current implementation forces a choice between:

• Using Optional (always downloads artifacts when job runs)
• No dependency at all (loses ordering guarantees)

This MR provides a third option: Optional_Artifacts for ordering-only dependencies.

Also check this MR !18701 (closed) where I use the Optional datatype to optionally read a dotenv file.

How

The implementation extends the CI type system with the following changes:

1. Type System Extension

(* Before *)
type need = {job : string; optional : bool}

(* After *)
type need = {job : string; optional : bool; artifacts : bool}

2. New Dependency Type

type dependency =
  | Job of tezos_job
  | Optional of tezos_job
  | Artifacts of tezos_job
  | Optional_Artifacts of tezos_job  (* New *)

3. Enhanced Dependency Resolution

The logic now considers both optional and artifacts flags:

if not optional then Some job
else if artifacts then Some job
else None

4. YAML Generation Updates

The YAML generation now outputs artifacts: false for optional dependencies that don't need artifacts.

Manually Testing the MR

1. Pipeline Structure Verification

# Check that the generated YAML includes artifacts: false for the converted jobs
grep -A 5 -B 5 "artifacts: false" .gitlab/ci/pipelines/before_merging.yml
grep -A 5 -B 5 "artifacts: false" .gitlab/ci/pipelines/merge_train.yml

2. Job Dependency Behavior

• Verify that oc.script:snapshot_alpha_and_link still runs correctly when:
  • oc.build_x86_64-released is present in the pipeline
  • oc.build_x86_64-released is absent from the pipeline
  • oc.build_x86_64-exp-dev-extra is present/absent

3. Artifact Download Verification

• Confirm that oc.script:snapshot_alpha_and_link does not download artifacts from the two converted jobs
• Check that job ordering is maintained (the job waits for the optional dependencies if they run)

This is a schduled pipeline : https://gitlab.com/tezos/tezos/-/pipelines/1930558124