excessive false positives from repository checks failing with dangling commits, tags, blogs
Summary
Support has been getting many more tickets for repository check failures since 15.0 when the feature was fixed for non-Geo sites.
In some cases, we're getting logs from git fsck which only show the following issues:
dangling tag 5c6886c774b713a43158aae35c4effdb03a3ceca
dangling blob 3e268c23fcd736db92e89b31d9f267dd4a50ac4b
dangling commit 919ff61d8d78c2e3ea9a32701dff70ecbefdd1d7GitLab automatically runs git gc and git repack on repositories after Git pushes:
- git gc runs a number of housekeeping tasks such as:
- Compressing Git objects to reduce disk space and increase performance.
- Removing unreachable objects that may have been created from changes to the repository, like force-overwriting branches.
I understand that these dangling tags/blobs/commits are examples of these unreachable objects, and so this issue should be resolved by garbage collection. This runs automatically, by default, every 200 pushes.
This is therefore not going to co-incide with repository checks, which as a result are creating lots of false positives
Our documentation is aware of this ..
If periodic repository checks cause false alarms, you can clear all repository check states
But we don't want to genrate false alarms. Folks shouldn't be sensitised to a high level of false positives, given that this feature exists to protect them from potential data loss scenarios.
similar/related
- #365745 (closed)
-
gitaly#2359 (moved)
- predates the 15.0 bug fix, but became a lot more active
Steps to reproduce
Example Project
What is the current bug behavior?
Excessive false positives for unreachable objects reported by repository check feature.
What is the expected correct behavior?
Improved signal:noise ratio
Approaches might include
- Running garbage collection before running the
fsck - Filtering out dangling objects from the output, and only returning a failure if there's anything else of concern
Relevant logs and/or screenshots
- Some attached to this support ticket - link for GitLab team members
- gitaly#2359 (comment 693225868)
- gitaly#2359 (comment 663521338)
At least one customer found: https://forum.gitlab.com/t/repocheck-dangling-commit/71256/9