Manual Gitaly node fail over

Problem to solve

If a failure makes a Gitaly node unavailable, there needs to be a way to promote a replica so that access continues.

We should focus on making the cross over process reliable and automatic, before we begin detecting failures automatically, since there is no point detecting a failure and failing over unreliably.

Further details

This should be the most naive implementation possible, so that we can put it in different situations, observe undesirable behavior and then address these situations iteratively.

Proposal

As an administrator

• from the Rails console I should be able to mark a repositories primary as unavailable
• GitLab should select a replica to promote to primary
• Git access should now occur from the secondary

Links / references