gitlab-ctl reconfigure inconsistently updates secrets from gitlab.rb to gitlab-secrets.json

Summary

gitlab-ctl reconfigure inconsistently handles secret token updates from gitlab.rb to gitlab-secrets.json. When both gitlab_shell['secret_token'] and gitaly['gitaly_token'] are defined in gitlab.rb, only the gitlab_shell token overwrites existing values in gitlab-secrets.json, while the gitaly token does not, causing a mismatch between the two secrets.

Steps to reproduce

1. Set up a GitLab instance with existing values in /etc/gitlab/gitlab-secrets.json for both:

   • gitlab_shell.secret_token
   • gitaly.gitlab_secret

2. Edit /etc/gitlab/gitlab.rb and set both tokens to new matching values:

   gitlab_shell['secret_token'] = 'new_matching_secret_token'
   gitaly['gitaly_token'] = 'new_matching_secret_token'

3. Run:

   gitlab-ctl reconfigure

4. Inspect /etc/gitlab/gitlab-secrets.json

Expected behavior

Both secrets in gitlab-secrets.json should be updated to match the values specified in gitlab.rb:

• gitlab_shell.secret_token → updated to new_matching_secret_token
• gitaly.gitlab_secret → updated to new_matching_secret_token

Actual behavior

Only one secret is updated:

• ✅ gitlab_shell.secret_token → correctly updated to new_matching_secret_token
• ❌ gitaly.gitlab_secret → retains old value, not updated from gitlab.rb

This creates a mismatch where the two secrets that should be identical are now different.

Additional context

Workaround discovered:

If both the gitlab_shell and gitaly keys are completely removed from gitlab-secrets.json before running reconfigure, then both values are successfully sourced from gitlab.rb. This confirms the inconsistency only occurs when overwriting existing values.

Why this matters:

When migrating Gitaly configurations or synchronizing secrets across nodes, administrators expect that values explicitly set in gitlab.rb will take precedence and be written to gitlab-secrets.json during reconfigure. The current inconsistent behavior makes it difficult to:

• Rotate secrets reliably
• Synchronize configurations across distributed Gitaly setups
• Troubleshoot authentication issues between GitLab Shell and Gitaly

Environment

• GitLab version: 18.4.4 (confirmed, likely affects earlier versions)
• Installation method: Linux package (Omnibus)
• Configuration: Standalone or distributed Gitaly setup

Relevant documentation

• Configure Gitaly - About the Gitaly token
• Configure Praefect - Requirements

Possible causes

The reconfigure logic appears to handle gitlab_shell['secret_token'] and gitaly['gitaly_token'] differently when writing to gitlab-secrets.json:

• gitlab_shell['secret_token'] → always overwrites existing gitlab-secrets.json values
• gitaly['gitaly_token'] → only writes if the key doesn't already exist in gitlab-secrets.json

This suggests different code paths or precedence rules for these two related secrets.