From ac45a1a2295ce4f0553c60f2154b2f9bee3985cb Mon Sep 17 00:00:00 2001 From: Daniel Diniz Date: Wed, 1 Oct 2025 11:25:48 +0000 Subject: [PATCH 1/3] Document scaling issue during Gitaly cluster migration --- doc/administration/gitaly/praefect/_index.md | 24 ++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 806fb39dc69874..016b497fbb3be7 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -412,6 +412,30 @@ To migrate to Gitaly Cluster (Praefect): Even if you don't use the `default` repository storage, you must ensure it is configured. [Read more about this limitation](../configure_gitaly.md#gitlab-requires-a-default-repository-storage). +### Storage move considerations for Kubernetes deployments + +{{< alert type="warning" >}} + +Known issue with HPA and storage moves: When using Horizontal Pod Autoscaler (HPA) with Sidekiq pods, mass storage moves can fail silently due to pod scaling during job execution. + +{{< /alert >}} + +Before performing mass storage moves, configure HPA with fixed replicas: + +```yaml +# Set minReplicas = maxReplicas to prevent scaling during migration +spec: + minReplicas: 4 + maxReplicas: 4 +``` + +If a (bulk) migration has already failed due to this issue, failed projects can remain stuck in a read-only state. +Affected repositories can be recovered via the following steps: + +1. [Reset affected projects to read-write](../../read_only_gitlab.md#make-the-repositories-read-only) state using `gitlab-rails console` +2. [Re-run storage moves via REST API](../../operations/moving_repositories.md) for individual projects +3. Restore HPA configuration after migration completes + ## Migrate off Gitaly Cluster (Praefect) If the limitations and tradeoffs of Gitaly Cluster (Praefect) are found to be not suitable for your environment, you can -- GitLab From 7c84b8ec5769066bd93d224656acaffa7f7c923c Mon Sep 17 00:00:00 2001 From: Daniel Diniz Date: Wed, 1 Oct 2025 11:30:14 +0000 Subject: [PATCH 2/3] Apply 1 suggestion(s) to 1 file(s) Co-authored-by: GitLab Duo --- doc/administration/gitaly/praefect/_index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 016b497fbb3be7..10bcc9309f66a3 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -432,7 +432,7 @@ spec: If a (bulk) migration has already failed due to this issue, failed projects can remain stuck in a read-only state. Affected repositories can be recovered via the following steps: -1. [Reset affected projects to read-write](../../read_only_gitlab.md#make-the-repositories-read-only) state using `gitlab-rails console` +1. [Reset affected projects to read-write state](../../read_only_gitlab.md#make-the-repositories-read-only) using `gitlab-rails console` 2. [Re-run storage moves via REST API](../../operations/moving_repositories.md) for individual projects 3. Restore HPA configuration after migration completes -- GitLab From 86535ed57067ab30fbc59d4e0bf90b2720574029 Mon Sep 17 00:00:00 2001 From: Daniel Diniz Date: Wed, 1 Oct 2025 11:40:28 +0000 Subject: [PATCH 3/3] Linting errors --- doc/administration/gitaly/praefect/_index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/administration/gitaly/praefect/_index.md b/doc/administration/gitaly/praefect/_index.md index 10bcc9309f66a3..1b745098eedfcc 100644 --- a/doc/administration/gitaly/praefect/_index.md +++ b/doc/administration/gitaly/praefect/_index.md @@ -433,8 +433,8 @@ If a (bulk) migration has already failed due to this issue, failed projects can Affected repositories can be recovered via the following steps: 1. [Reset affected projects to read-write state](../../read_only_gitlab.md#make-the-repositories-read-only) using `gitlab-rails console` -2. [Re-run storage moves via REST API](../../operations/moving_repositories.md) for individual projects -3. Restore HPA configuration after migration completes +1. [Re-run storage moves via REST API](../../operations/moving_repositories.md) for individual projects +1. Restore HPA configuration after migration completes ## Migrate off Gitaly Cluster (Praefect) -- GitLab