FIPS Runners fail to start jobs with gitlab-runner 18.2.1 (current latest), give the error "panic: curve25519: internal error: scalarBaseMult was not 32 bytes"
Summary
Our FIPS runners will fail with any GitLab Runner version higher than 18.0.5 and throws a go panic with the latest 18.2.1 version
Steps to reproduce
Note, this configuration works fine with 18.0.5 GitLab runner version.
Have a FIPS runner at version 18.2.1 with the configuration as follows:
concurrent = 256
check_interval = 5
log_format = "json"
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "ip-1-1-1-185"
url = "REDACTED"
id = 4
token = "glrt-REDACTED"
executor = "docker-autoscaler"
[runners.docker]
#host = "unix:///var/run/docker.sock"
image = "ubuntu:latest"
helper_image_flavor = "ubi-fips"
pull_policy = "if-not-present"
network_mode = "host"
[runners.autoscaler]
plugin = "aws:latest"
instance_ready_command = "sudo cloud-init status --wait"
[runners.autoscaler.plugin_config]
name = "worker"
[runners.autoscaler.connector_config]
username = "username"
[[runners.autoscaler.policy]]
idle_count = 0
idle_time = "5m0s"If we use version 18.1.0 we hit this bug which is solved but I believe it is related #38890 (closed)
If you update to 18.2.1 (currently the latest available) and run gitlab-runner verify you get this error:
# gitlab-runner verifyINFO[0000] FIPS mode is enabled. Using an external SSL library.
Runtime platform arch=amd64 os=linux pid=4434 revision=cc489270 version=18.2.1
Running in system-mode.
There might be a problem with your config based on jsonschema annotations in common/config.go (experimental feature):
jsonschema: '/runners/0/Autoscaler/Policy/0/PreemptiveMode' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/Autoscaler/$ref/properties/Policy/items/$ref/properties/PreemptiveMode/type: expected boolean, but got null
Verifying runner... is valid correlation_id=01K239Q6JG2RT1T2FGD20JDZEF runner=yFuKOFdTiAdding preemptive_mode = false to the configuration will get you past this error and gitlab-runner verify will pass. However trying to run a job the GitLab runner logs will give the following error:
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: panic: curve25519: internal error: scalarBaseMult was not 32 bytes
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: goroutine 348 [running]:
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/curve25519.ScalarBaseMult(0xc0018c0660, 0x40012a0?)
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/curve25519/curve25519.go:39 +0x13b
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/ssh.(*curve25519KeyPair).generate(0xc0018c0640, {0x400c8a0?, 0x40012a0?})
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/kex.go:457 +0x65
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/ssh.(*curve25519sha256).Client(0x63376a0?, {0x7faa3d271d48, 0xc001903b00}, {0x400c8a0, 0x40012a0}, 0xc001703ec0)
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/kex.go:468 +0x6e
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/ssh.(*handshakeTransport).client(0xc000e40820, {0x402aaa0?, 0x64f8c20?}, 0xc001703ec0?)
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/handshake.go:826 +0x66
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/ssh.(*handshakeTransport).enterKeyExchange(0xc000e40820, {0xc000f81680, 0x405, 0x405})
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/handshake.go:716 +0x49a
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: golang.org/x/crypto/ssh.(*handshakeTransport).kexLoop(0xc000e40820)
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/handshake.go:345 +0x98
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: created by golang.org/x/crypto/ssh.newClientTransport in goroutine 313
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS gitlab-runner[4466]: /go/pkg/mod/golang.org/x/crypto@v0.39.0/ssh/handshake.go:170 +0x256
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS systemd[1]: gitlab-runner.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 07 22:20:38 ip-REDACTED_IP_ADDRESS systemd[1]: gitlab-runner.service: Failed with result 'exit-code'.
.gitlab-ci.yml
stages:
- helloWorld
helloWorld:
stage: helloWorld
script:
- echo "Hello world."Actual behavior
The job fails to start and the GitLab runner throws a go panic in the logs
Expected behavior
I expect the job to start and run
Relevant logs and/or screenshots
job log
The job fails to start at all and hangs with this output:Running with gitlab-runner 18.2.1 (cc489270)
on worker yFuKOFdTi, system ID: s_7cca2598381a
feature flags: FF_TIMESTAMPS:true
Resolving secrets
Preparing the "docker-autoscaler" executor
...Environment description
This is a FIPS compliant self hosted ommnibus GitLab on version 18.1.1 The runners are also self hosted
concurrent = 256
check_interval = 5
log_format = "json"
log_level = "info"
[session_server]
session_timeout = 1800
[[runners]]
name = "ip-1-1-1-185"
url = "REDACTED"
id = 4
token = "glrt-REDACTED"
executor = "docker-autoscaler"
[runners.docker]
#host = "unix:///var/run/docker.sock"
image = "ubuntu:latest"
helper_image_flavor = "ubi-fips"
pull_policy = "if-not-present"
network_mode = "host"
[runners.autoscaler]
plugin = "aws:latest"
instance_ready_command = "sudo cloud-init status --wait"
[runners.autoscaler.plugin_config]
name = "worker"
[runners.autoscaler.connector_config]
username = "username"
[[runners.autoscaler.policy]]
idle_count = 0
idle_time = "5m0s"Used GitLab Runner version
Possible fixes
Edited by Constantine Mavromichalis