[go: up one dir, main page]

[SPIKE] Pull Mirroring - Investigate if LFS objects can be sync'd before git data written

Overview

In #422486 (comment 1780284807) I documented a scenario where if you run a git clone of a repo that has LFS objects before the LFS objects have completely sync'd via pull mirroring, you can experience errors such as Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it:

$ rm -rf lfs-mirror-test2 ; git clone http://gdk.test:3000/root/lfs-mirror-test2.git
Cloning into 'lfs-mirror-test2'...
--snip--
Downloading lfs.text (5 B)
Error downloading object: lfs.text (c9d04c9): Smudge error: Error downloading lfs.text (c9d04c9565fc665c80681fb1d829938026871f66e14f501e08531df66938a789): [c9d04c9565fc665c80681fb1d829938026871f66e14f501e08531df66938a789] Object does not exist on the server or you don't have permissions to access it: [404] Object does not exist on the server or you don't have permissions to access it

But re-attempting the same git clone command a few moments later succeeds because by the time, all LFS objects have sync'd.

$ rm -rf lfs-mirror-test2 ; git clone http://gdk.test:3000/root/lfs-mirror-test2.git
Cloning into 'lfs-mirror-test2'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 0), reused 7 (delta 0), pack-reused 0
Receiving objects: 100% (7/7), done.

Proposal

Investigate if we can sync LFS objects before writing git data when pull mirroring.

Spikes

'Temporary branch method'

Draft: Temporary branches method [SPIKE] Pull M... (!164033 - closed) explores using a temporary branch and updating it first (e.g. update-mirror/main-tmp), downloading LFS objects from the temporary branch and then applying commits from update-mirror/main-tmp to update-mirror/main.

Updating Gitaly project.repository.fetch_mirror() to accept a check_repo_changed: true argument

Using keep-around refs

Abandoned.

cc @mcbabin

Edited by Ash McKenzie