Bug in Gitlab Restore since 17.11

General
,
1

Hi all,

I have a self-managed Gitlab instance which recently updated to v17.11. This instance is synced daily on a secondary geolocation for disaster recovery purposes, using the native gitlab-backup tool.

My problem : since the update from 17.10 to 17.11, the gitlab-restore tool is failing when restoring the database:

…
ALTER TABLE
REVOKE
GRANT
rake aborted!
NoMethodError: undefined method chomp' for nil:NilClass /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/task_helpers.rb:65:in prompt’
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/task_helpers.rb:30:in ask_to_continue' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/restore/process.rb:55:in output_warning’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/restore/process.rb:35:in execute!' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:101:in run_restore_task’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:168:in block in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:165:in each_value’
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:165:in run_all_restore_tasks' /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:68:in restore’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:21:in block in restore_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:75:in lock_backup’
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:18:in restore_backup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:123:in block (3 levels) in <top (required)>’
/opt/gitlab/embedded/bin/bundle:25:in load' /opt/gitlab/embedded/bin/bundle:25:in ’
Tasks: TOP => gitlab:backup:restore
(See full trace by running task with --trace)
2025-04-24 07:47:19 UTC – ------ BEGIN ERRORS -----
2025-04-24 07:47:19 UTC – ERROR: must be able to SET ROLE “gitlab-psql”
2025-04-24 07:47:19 UTC – ------ END ERRORS -------
2025-04-24 07:47:19 UTC – [DONE]
2025-04-24 07:47:19 UTC – Source backup for the database ci doesn’t exist. Skipping the task
2025-04-24 07:47:19 UTC – Restoring database … done
2025-04-24 07:47:19 UTC – There were errors in restoring the schema. This may cause issues if this results in missing indexes, constraints, or columns. Please record the errors above and contact GitLab

What I would need from this community :
- has somebody encoutered the same problem ?
- what informations should I gather in order to open a workable ticket for the Gitlab Team ?

Thank you in advance for your help !

Max

2

We are experiencing the same problem. In our case we did update Gitlab to 17.11 (Postgres automatically got updated to version 16)

EDIT:
Managed to solve the issue by manually executing:

$ gitlab-psql
> GRANT "gitlab-psql" TO gitlab;
1 Like
3

I’m not sure, but I don’t think it’s a good idea to increase the rights of the user “gitlab-psql” to the same level as the user “gitlab”, in my opinion this gives the user too many authorizations.

Have a look at this Issue: Update Legacy Rake Restore Task to Ignore New Postgresql Error (#542965) · Issues · GitLab.org / GitLab · GitLab, so hopefully it should be fixed soon.

1 Like
4

Thanks for linking that issue, bnburger. That makes sense, granting extra permissions always feels like a workaround that could open up new problems. I’ll keep an eye on that GitLab issue thread and hope they roll out a proper fix soon.

1 Like
5

Hi guys, thank you all for your answers !
I did the temporary workaround too, but I’ll keep an eye on the ticket resolution.

6

This error just occured for me. Trying to transfer my gitlab-ce self hosted to a new server. It happened in the middle of backup restore. I told it to continue/ignore. I have no idea the consequences but I had no idea what the consequences would be for stopping/trying to adjust user permissions in the middle of a script.

7

Can anyone figure out when this fix is going out, in what release that is ?

I see merge request Force rake gitlab:backup:restore to ignore PG16 must be able to SET ROLE “gitlab-psql” error was merged into master, but I’ve reviewed changelog for 18.0.1 and there is no mention of that merge request inside the changelog.

Usually i cannot figure out their release patterns so i have no idea if this fix made it into a release or not.

8

According to the merge request you linked, the milestone says 18.1. As 18.0.1 is an 18.0 release and not an 18.1 release, then it means it hasn’t been released yet. Assuming of course that it has made the milestone.

According to here: GitLab releases | GitLab 18.1 will be released June 19 so in a few days we shall know for sure.

9

Thanks, i did notice that but figured since it says missed i was unsure what to expect. But yeah, i guess we will find out in about few days.

1 Like
10

It’s possible they missed the deadline, I’ve no idea what the cut-off period is for a release. Last week when it was merged may have been too late, so it could well appear in a minor release of 18.1.x or at worst in 18.2.

11

Hey everyone,

The 18.4 update came with another problem (amcheck extension can’t be installed by gitlab user), so ChatGPT proposed a more global solution, that I recommend:

# give gitlab superuser role
gitlab-psql -d postgres -c "ALTER ROLE gitlab WITH SUPERUSER;"

# start the restore process
gitlab-backup restore

# remove superuser role
gitlab-psql -d postgres -c "ALTER ROLE gitlab WITH NOSUPERUSER;"

I believe this to be the more sustainable solution, this is why I’m pinning this answer.
But thanks to @Motii1 for the original fix !

2 Likes