[go: up one dir, main page]
Skip to content

External diffs stored on local storage are not migrated to object storage

Summary

There doesn't appear to be a built-in process to migrate external diffs from local storage to object storage, and nor is there a documented manual process.

While diffs stored in the database are automatically migrated to local or object storage (depending on the configuration) when gitlab_rails['external_diffs_enabled'] = true is set (in batches of 1000 via the cronjob:schedule_migrate_external_diffs sidekiq queue), if you have already migrated diffs to local storage and then switch to object storage the files on local storage and left as is.

This is a problem for multi-node installations where the files will only be accessible to a single rails node, unless an NFS share is used. And using NFS goes against the goal of replacing the need for NFS with object storage.

We should either:

  • provide a method of automatic migration of diffs from local to object storage
  • provide a rake task to migrate the files
  • document a manual process for transferring the files to object storage and updating the database
  • document that files already on local storage will not be migrated, and in a multi-node environment an NFS share will need to be maintained to make the local files available to all nodes

Steps to reproduce

  1. Configure gitlab to store external diffs on local disk (gitlab_rails['external_diffs_enabled'] = true).
  2. Generate some diffs.
  3. Configure external diffs object storage (gitlab_rails['external_diffs_object_store_enabled'] = true).
  4. Confirm diffs still on local storage and not on object storage.
  5. Try to access the diffs page for an MR from another rails node. The page will fail to load and a "No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/external-diffs/merge_request_diffs/mr-9026/diff-18493" 500 error will be logged.

What is the current bug behavior?

Diffs cannot be accessed from all rails nodes unless an NFS share is maintained for /var/opt/gitlab/gitlab-rails/shared/external-diffs

What is the expected correct behavior?

A process should be available to migrate external diffs files from local to object storage as there is for other object types.

Results of GitLab environment info

Tested in version 14.10.3.

Possible fixes

A workaround might be to manually copy files from /var/opt/gitlab/gitlab-rails/shared/external-diffs to the object storage bucket, and then run the gitlab-rake gitlab:external_diffs:force_object_storage to update the database.