GitLab Runner FIPS default image flavor is not ubi-fips
Summary
When gitlab-runner is installed in FIPS mode on an Internet disconnected system using the gitlab-runner-fips RPM versions >=17.3.2, 17.4.x, and 17.5.x the prebuilt image is not used out-of-the-box. We've been experiencing this issue since the upgrade from 17.3.1 to any higher version. The current version we're running and experiencing the issue still is 17.5.2.
Steps to reproduce
Install gitlab-runner-fips 17.3.2, 17.4.x, or 17.5.x. Attempt to run any job.
Actual behavior
Job fails to run because it cannot pull the helper image.
Expected behavior
Job succeeds by using the prebuilt helper image.
Relevant logs and/or screenshots
runner debug log
...
Looking for prebuilt image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2... job=2135 project=3 runner=xxxxxx-TS
Failed to load prebuilt image from: /usr/bin/out/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /usr/bin/out/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory job=2135 project=3 runner=xxxxxx-TS
Failed to load prebuilt image from: /usr/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /usr/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory job=2135 project=3 runner=xxxxxx-TS
Failed to load prebuilt image from: /home/gitlab-runner/out/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /home/gitlab-runner/out/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory job=2135 project=3 runner=xxxxxx-TS
Failed to load prebuilt image from: /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz error: open /usr/lib/gitlab-runner/helper-images/prebuilt-alpine-x86_64.tar.xz: no such file or directory job=2135 project=3 runner=xxxxxx-TS
Using helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2 job=2135 project=3 runner=xxxxxx-TS
Loading image form registry: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2 job=2135 project=3 runner=xxxxxx-TS
Looking for image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2 ... job=2135 project=3 runner=xxxxxx-TS
Loaded Docker credentials, source = "$DOCKER_AUTH_CONFIG", hostnames = [], error = <nil> job=2135 project=3 runner=xxxxxx-TS
Loaded Docker credentials, source = "job payload (GitLab Registry)", hostnames = [git.my.domain:443], error = <nil> job=2135 project=3 runner=xxxxxx-TS
No credentials found for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2 job=2135 project=3 runner=xxxxxx-TS
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.2 ... job=2135 project=3 runner=xxxxxx-TS
...Environment description
Internet disconnected environment on FIPS-compliant RHEL 9 host using the docker executor setup to work with podman.
config.toml contents
concurrent = 20
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "git-runner"
url = "https://git.my.domain"
id = 1
token = "xxxx-xxxxxx-xxxxxxxxxx-xx"
token_obtained_at = 2024-06-28T20:16:02.048850
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
tls-ca-file = "/etc/pki/tls/cert.pem"
[runners.custom_build_dir]
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.docker]
# Adding this option fixes the issue
# helper_image_flavor = "ubi-fips"
tls_verify = false
image = "c2net:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
network_mtu = 0
host = "unix:///run/user/981/podman/podman.sock"
pull_policy = "if-not-present"Used GitLab Runner version
Version: 17.5.2
Git revision: c6eae8d7
Git branch: 17-5-stable
GO version: go1.22.7 X:boringcrypto
Built: 2024-10-22T14:54:09+0000
OS/Arch: linux/amd64Possible fixes
If the helper_image_flavor option is set to ubi-fips in the config.toml file, the prebuilt image is used as expected. Would be helpful to have this spelled out in the documentation for the FIPS installation if the default is not adjusted.
Likely needs a condition for the FIPS version to ensure ubi-fips is used as the default flavor instead of alpine for the FIPS package since the only flavor image shipped with the FIPS version is ubi-fips.