Bug in Remove Blobs causes error "source repository checksum altered"

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

When attempting to run the "Remove blobs" feature from the UI, the operation fails with an error "source repository checksum altered" and no blobs are actually removed.

Steps to reproduce

Try running the "Remove blobs" from the UI.

What is the current bug behavior?

The error stops the "remove blobs" from actually removing anything.

What is the expected correct behavior?

That the blobs get cleaned up.

Relevant information

1. The process is async, Sidekiq starts RewriteHistoryWorker which results in RewriteHistoryService.
2. RewriteHistoryService sets repository to read only state here (makes sense, we don't want changes to be pushed in the middle of the history rewrite).
3. That service communicates with Gitaly calling rewrite_history method here #repository_cleaner
4. Gitaly starts git-filter-repo and runs this function #rewrite_history
5. Gitaly takes a checksum of the repo and makes a copy of the repo
6. Gitaly performs history rewrite/blob removal on a copy of the repo
7. After rewrite is done, Gitaly take a checksum of the original repo one more time
8. Gitaly compares both checksums (remember, according to this line the repo should be in read only state)
9. However, the error that gets returned is source repository checksum altered

The customer tried this 3 times in a row with the same result.

Output of checks

Results of GitLab environment info

GitLab version 17.8.5, Helm installation

Note: This information is brought in by a customer in ticket https://gitlab.zendesk.com/agent/tickets/612861 (internal only)