CI: attempt to fix OOMs on arm64 runners in the Debian packaging pipelines
This is one more symptom of a problem with all the jobs/runners using ramfs. Since we mount the entire build space in ram, depending on the nature of the job, the artifacts generated, the size of the cache ( if present ), the targets of the makefile, etc, Sometimes the memory is saturated.
This MR is an attempt to fix the problem for this specific job, assuming that the problem is that on arm64 the space ( and ram ) is smaller than amd64 runners and creating to many artifacts leads to OOMs regularly. It is a follow-up of !19259 (merged).
This MR does not address the underlying problem, that is we run our jobs in a constrained environment ( cpu/ram ), but our jobs are not constrained at all and can potentially overuse the available resources and fail.
What
- Reduce memory pressure during Debian packaging on arm64 by:
- Moving built binaries into the packaging area instead of copying them (cp -> mv) in scripts/packaging/build-deb-local.sh.
- Performing a pre-build cleanup to avoid leftover artifacts increasing the working set.
Why
- Debian packaging jobs on arm64 runners have been hitting OOM conditions. The first attempt mitigated some pressure but did not fully address spikes during the packaging stage. Moving large executables instead of duplicating them, and cleaning the tree before the build, reduces peak memory and I/O.
How
- scripts/packaging/build-deb-local.sh:
-
cp -f "$ex"… →mv -f "$ex"… to avoid duplicating large files in memory/storage. - Add a cleanup step before dpkg-buildpackage:
- make clean (currently runs on local builds; CI path remains unchanged).
-
Test
check this scheduled pipeline : https://gitlab.com/tezos/tezos/-/pipelines/2047233965 focusing on the build jobs
notice, in my last attempt, the scheduled pipeline was green, but it didn't fix the problem.