[go: up one dir, main page]

CI: use spot high_cpu nodes in some key jobs to speed up pipelines

What

Optimize the pipeline to decrease overall runtime, with a special focus on the before_merging type which is the most frequent.

Why

Have a great user experience as a contributor and increase iterations so productivity.

How

Tests were conducted, and the results showed that switching to instances with a 1:1 CPU-to-RAM ratio reduced the walltime from 44 minutes to 30 minutes on some critical paths.

At the same time, the cost related to these execution will be divided by two using the spots instances.

See details below:

image

Then following jobs need to be tagged with gcp_high_cpu:

  • oc.unit:proto-x86_64
  • oc.unit:other-x86_64
  • oc.unit:protocol_compiles
  • ocaml-check
  • build-layer1-profiling
  • oc.build_x86_64-exp-dev-extra
  • oc.build_x86_64-released
  • oc.build:static-x86_64-linux-binaries

Update (Feb 7, 2025): We've broken the action plan into three steps, this MR implements the first:

  • Step 1 (Implemented in this MR): Run the following jobs on the gcp_high_cpu tag (n2d-highcpu-16 VMs): oc.unit:proto-x86_64, oc.unit:other-x86_64, oc.unit:protocol_compiles, ocaml-check, oc.build_x86_64-exp-dev-extra, and oc.build_x86_64-released (see !16340 (merged)).
  • Step 2 (Future): Set up a new k8s node pool gcp_high_cpu_extended based on machine type n2d-highcpu-32.
  • Step 3 (Future): Execute jobs oc.build:static-x86_64-linux-binaries and build-layer1-profiling on node pool gcp_high_cpu_extended (n2d-highcpu-32 VMs).
Edited by Neo

Merge request reports

Loading