Migrate housekeeping scheduler to Gitaly

Status

We need this before Raft replication on production.

Background

Recently, a majority of housekeeping tasks have been triggered by Gitlab Rails. GitLab Rails maintains a push counter. Whenever a Git push is successful, it increases this counter. It triggers an OptimizeRepository() when the counter exceeds a configurable threshold. The instance administrator can update this configuration accordingly.

At Gitaly's side, the corresponding handler uses a sort of heuristics to determine which tasks to perform and the configurations of each task. The overall heuristic is quite complicated. It inspects the state of the repository carefully, including loose refs count, packfile structure, last modification, etc. Apparently, housekeeping is expensive and we don't want to perform a full run too frequently. Thus, the housekeeping manager attempts to balance between costs and benefits of optimization. This heuristic can be found here.

Whenever a housekeeping task runs, Gitaly needs to perform an inspection to determine the repository state. That's because Gitaly doesn't persist any metadata related to the prior housekeeping run. In addition, there is no centralized writer to account for such information accurately. It's very common for a task to run prematurely or too late. The former case makes the repository not optimized until the next trigger. The later requires a more expensive strategy which can add up the latency.

Although most of the heavy part is handled by Gitaly, the scheduling part is still under the control of GitLab Rails. With this current scheduling scheme, Gitaly is put in a reactive position instead of triggering the housekeeping process proactively. So, it's troublesome if Gitlab Rails doesn't trigger that RPC frequently enough or doesn't trigger at all. One typical example is gitlab-com/gl-infra/scalability#2547.

When Gitaly's Write-ahead logging is applied, all writes are synchronized by a writer. We also introduce a file-based database to persist WAL-related information. That makes housekeeping accounting more convenient. Besides, along the way, housekeeping needs a big touch-up to work with WAL.

This issue proposes to move the Housekeeping scheduler to Gitaly. After each write operation, the transaction manager updates the accounting information and triggers a subset of the task correspondingly. This change improves the timing of housekeeping, making it run more on-time and precisely. It also removes the housekeeping frequency setting and simply the system a bit.