[Feature flag] Enable housekeeping middleware

IMPORTANT: when Gitaly is started in a Rails test, all feature flags are enabled: https://gitlab.com/gitlab-org/gitlab/blob/28bdb35c83fbfcfca09a9d237bfecb71e55b8f9e/spec/support/helpers/gitaly_setup.rb#L69

Any flags which change user-facing behaviour have a chance of causing test failures, which in turn will block Gitaly deployment MRs. In these situations, you can do the following:

1. Get the Gitaly change reviewed and merged. Validate the Gitaly pipeline is passing on the default branch.
2. Open a Rails MR that simultaneously updates the commit SHA in GITALY_SERVER_VERSION (https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITALY_SERVER_VERSION) and contains the updated assertions. This will allow tests to execute against a version of Gitaly that contains your changes, while updating the version at the same time.
3. Get the Rails MR reviewed and merged.

What

Enable the :housekeeping_middleware feature flag. The flag controls whether the middleware introduced in this MR will execute. The middleware tracks the number of write RPCs that run against each repository and decides whether an OptimizeRepository housekeeping task should be run with the heuristical strategy.

The middleware itself should add very little overhead as housekeeping tasks are executed asynchronously. The middleware maintains a persistent map for bookkeeping, but impact on memory should be minimal.

Owners

• Team: Gitaly
• Most appropriate slack channel to reach out to: #g_gitaly
• Best individual to reach out to: @jamesliu-gitlab

Expectations

What release does this feature occur in first?

18.1

What are we expecting to happen?

When the flag is enabled, all incoming gRPC requests will be routed through an additional gRPC middleware. The middleware does not alter the request in any way, and will concurrently start a housekeeping task if required. The middleware is independent of transactions.

What might happen if this goes wrong?

• Errors encountered in the middleware do not affect the outcome of the RPC.
• More housekeeping operations may impose a heavy load on Gitaly nodes.

What can we monitor to detect problems with this?

https://dashboards.gitlab.net/d/gitaly-main/gitaly3a-overview?orgId=1&from=now-6h/m&to=now/m&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main

Roll Out Steps

• Enable on staging
  • Is the required code deployed on staging? (howto)
  • Enable on staging (howto)
  • Add featureflagstaging to this issue (howto)
  • Test on staging (howto)
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Enable on production
  • Is the required code deployed on production? (howto)
  • Progressively enable in production (howto)
  • Add featureflagproduction to this issue
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Create subsequent issues
  • To default enable the feature flag (optional, only required if backwards-compatibility concerns exist)
    • Create issue using the Feature Flag Default Enable template.
    • Set milestone to current+1 release
  • To Remove feature flag
    • Create issue using the Feature Flag Removal template.
    • Set milestone to current+1 (+2 if we created an issue to default enable the flag).

Please refer to the documentation of feature flags for further information.