[go: up one dir, main page]

backup: Add support for incremental server-side backups

James Liurequested to merge
jliu-server-side-incremental-backups into master

Context

#5461 (closed)

Functionality to perform incremental backups exists, but is not yet invoked when executing server-side backups. This is because server-side backups are initiated via an RPC call which didn't include incremental as a request option.j

This MR modifies the BackupRepositoryRequest protobuf message to include the incremental flag, and passes it through to the backup manager on the other end. Surrounding tests have been modified to exercise this option, but no net-new tests have been added to explicitly check that incremental is being passed correctly.

Testing

This can easily be tested locally via GDK.

  1. Enable object storage by editing gdk.yml:
object_store:
  enabled: true
$ gdk reconfigure && gdk restart
  1. Check that minio is running correctly. It listens on port 9002, but the hostname will differ depending on how you've configured DNS in GDK. For me, it runs at http://gdk.test:9002. Log in with username minio and password gdk-minio.
  2. Create a new bucket named gitaly-backups.
  3. Create a full server-side backup using the Rake task and note the computed backup ID:
$ cd gitlab
$ bundle exec rake gitlab:backup:create SKIP=db REPOSITORIES_SERVER_SIDE=true
  1. Browse to gitaly-backups/manifests/default/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git (this SHA corresponds to the gitlab-org/gitlab-test default project) and note the single toml file. It should look something like this:
object_format = 'sha1'

[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.custom_hooks.tar'
  1. Make a small change to the gitlab-org/gitlab-test project, e.g. adding a new file.
  2. Now, create an incremental server-side backup using the Rake task, using the previously computed backup ID as the baseline:
$ bundle exec rake gitlab:backup:create SKIP=db REPOSITORIES_SERVER_SIDE=true INCREMENTAL=yes PREVIOUS_BACKUP=1697498176_2023_10_16_16.5.0-pre
  1. Again, browse to gitaly-backups/manifests/default/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git and note an additional toml file. The file should contain two steps and look something like this:
object_format = 'sha1'

[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.custom_hooks.tar'

[[steps]]
bundle_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.bundle'
ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.refs'
previous_ref_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/001.refs'
custom_hooks_path = '@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35/1697498176_2023_10_16_16.5.0-pre/002.custom_hooks.tar'

The additional step confirms that an incremental backup of the gitlab-org/gitlab-test repository has taken place. Since the backup artifacts were uploaded to object storage, this also confirms server-side backups were used.

Edited by James Liu

Merge request reports