From 7e8e766275577f33f0f39492d70d9516bf35e5a6 Mon Sep 17 00:00:00 2001 From: Sami Hiltunen Date: Tue, 31 Jan 2023 20:00:30 +0200 Subject: [PATCH] Document invalid metadata deletions being default enabled Praefect has default enabled invalid metadata deletion. Document this fact. --- doc/administration/gitaly/praefect.md | 17 ++++++++++------- doc/update/index.md | 12 ++++++++++++ 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 3f1c9be39853fa..f54f64227b336a 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1421,22 +1421,25 @@ praefect['configuration'] = { #### Enable deletions +> - [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4080) and disabled by default in GitLab 15.0 +> - [Default enabled](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/5321) in GitLab 15.9. + WARNING: -Deletions are disabled by default due to a race condition with repository renames that can cause incorrect -deletions. This is especially prominent in Geo instances as Geo performs more renames than instances without Geo. -You should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. +Deletions were disabled by default prior to GitLab 15.9 due to a race condition with repository renames +that can cause incorrect deletions. This is especially prominent in Geo instances as Geo performs more renames +than instances without Geo. In GitLab 15.0 to 15.5, you should enable deletions only if the [`gitaly_praefect_generated_replica_paths` feature flag](index.md#praefect-generated-replica-paths-gitlab-150-and-later) is enabled. The feature flag was removed in GitLab 15.6 making deletions always safe to enable. -By default, the worker does not delete invalid metadata records but logs them and outputs Prometheus -metrics for them. +By default, the worker deletes invalid metadata records. It also logs the deleted records and outputs Prometheus +metrics. -You can enable deleting invalid metadata records with: +You can disable deleting invalid metadata records with: ```ruby praefect['configuration'] = { # ... background_verification: { # ... - delete_invalid_records: true, + delete_invalid_records: false, }, } ``` diff --git a/doc/update/index.md b/doc/update/index.md index ebed90809e85fd..ea203dfceb7622 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -268,6 +268,18 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap - This version removes `SanitizeConfidentialTodos` background migration [added](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87908/diffs) in 15.6, which removed any user inaccessible to-do items. Make sure that this migration is finished before upgrading to 15.9. - As part of the [CI Partitioning effort](../architecture/blueprints/ci_data_decay/pipeline_partitioning.md), a [new Foreign Key](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/107547) was added to `ci_builds_needs`. On GitLab instances with large CI tables, adding this constraint can take longer than usual. Make sure that this migration is finished before upgrading to 15.9. +- Praefect's metadata verifier's [invalid metadata deletion behavior](../administration/gitaly/praefect.md#enable-deletions) is now enabled by default. + + The metadata verifier processes replica records in the Praefect database and verifies the replicas actually exist on the Gitaly nodes. If the replica doesn't exist, its + metadata record is deleted. This enables Praefect to fix situations where a replica has a metadata record indicating it's fine but, in reality, it doesn't exist on disk. + After the metadata record is deleted, Praefect's reconciler schedules a replication job to recreate the replica. + + Because of past issues with the state management logic, there may be invalid metadata records in the database. These could exist, for example, because of incomplete + deletions of repositories or partially completed renames. The verifier deletes these stale replica records of affected repositories. These repositories may show up as + unavailable repositories in the metrics and `praefect dataloss` sub-command because of the replica records being removed. If you encounter such repositories, remove + the repository using `praefect remove-repository` to remove the repository's remaining records. + + You can find repositories with invalid metadata records prior in GitLab 15.0 and later by searching for the log records outputted by the verifier. You can find an example log record [here](../administration/gitaly/praefect.md#repository-verification). - Praefect configuration structure in Omnibus GitLab [has changed](https://gitlab.com/gitlab-org/gitaly/-/issues/4467) to be consistent with the Praefect configuration structure used in source installs. Praefect configuration is now under `praefect['configuration']` as a single hash. There are still other top-level keys in `praefect` used by Omnibus GitLab. -- GitLab