From 334933b9eaa58480482dfff6cc03b8f905d59c3a Mon Sep 17 00:00:00 2001 From: Ben Prescott Date: Thu, 30 Sep 2021 19:50:06 +0100 Subject: [PATCH 1/4] Clarify 14.1 on the upgrade path --- doc/update/index.md | 116 ++++++++++++++++++++++++++++---------------- 1 file changed, 75 insertions(+), 41 deletions(-) diff --git a/doc/update/index.md b/doc/update/index.md index 7f9ded58d932da..65e0c2fc4ce603 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on. Make sure to read the whole page as it contains information related to every upgrade method. +NOTE: +Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`. +This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may +be fixes for issues relating to the upgrade process. + The [maintenance policy documentation](../policy/maintenance.md) has additional information about upgrading, including: @@ -76,22 +81,45 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ## Checking for background migrations before upgrading -Certain major/minor releases may require different migrations to be -finished before you update to the newer version. +Certain releases may require different migrations to be +finished before you update to the newer version. Additionally check +[batched migrations](#batched-background-migrations) from GitLab 14.0. Decrease the time required to complete these migrations by increasing the number of [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -**For GitLab 14.0 and newer** +**For Omnibus installations** + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +**For installations from source** + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +### Batched background migrations + +GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md). + +NOTE: +Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade. -To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): +#### Status check + +To check the status of batched background migrations: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Migrations**. ![queued batched background migrations table](img/batched_background_migrations_queued_v14_0.png) +All migrations must have a `Finished` status before you upgrade GitLab. + The status of batched background migrations can also be queried directly in the database. 1. Log into a `psql` prompt according to the directions for your instance's installation method @@ -102,20 +130,14 @@ The status of batched background migrations can also be queried directly in the select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; ``` -**For Omnibus installations** - -You can also run: +If the migrations are not finished and you try to update to a later version, +you'll see an error like: -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +```plaintext +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` -**For installations from source** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -``` +[Read about the options for completing an upgrade if this happens](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). ### What do I do if my background migrations are stuck? @@ -148,6 +170,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` +**Batched migrations (GitLab 14.0 and newer)** + +[Read more about troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). + ## Dealing with running CI/CD pipelines and jobs If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. @@ -197,7 +223,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.12`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -320,6 +346,11 @@ Git 2.33.x and later is required. We recommend you use the ### 14.3.0 +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). + +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.3.Z from earlier GitLab 14 releases. + - Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) for how to proceed. @@ -343,11 +374,11 @@ for how to proceed. ### 14.2.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to 14.2. +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). + +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.2.Z from earlier GitLab 14 releases. + - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) @@ -372,35 +403,26 @@ for how to proceed. ### 14.1.0 -Due to an issue where `BatchedBackgroundMigrationWorkers` were -[not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) -for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) -and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need -to update to at least 14.1.0 that contains the same fix before you update to -a later version. - -After you update to 14.1.0, -[batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) -before you update to a later version. - -If the migrations are not finished and you try to update to a later version, -you'll see an error like: - -```plaintext -Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': -``` +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) + but can upgrade to 14.1.Z. -See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). +- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. + 14.1 is included on the upgrade path for the broadest compatibility + with self-managed installations, and ensure 14.0.0-14.0.4 installations do not + encounter issues with [batched background migrations](#batched-background-migrations). ### 14.0.0 +- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher. + - Due to an issue where `BatchedBackgroundMigrationWorkers` were [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - that requires an update to at least 14.0.5. + that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). After you update to 14.0.5 or a later 14.0 patch version, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) + [batched background migrations need to finish](#batched-background-migrations) before you update to a later version. If the migrations are not finished and you try to update to a later version, @@ -419,6 +441,18 @@ See how to [resolve this error](../user/admin_area/monitoring/background_migrati You should instead follow a [supported upgrade path](#upgrade-paths). - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. +#### Upgrading to later 14.Y releases + +- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, + because of [batched background migrations](#batched-background-migrations). + + 1. Upgrade first to either: + - 14.0.5 or a later 14.0.Z patch release. + - 14.1.0 or a later 14.1.Z patch release. + + 1. [Batched background migrations need to finish](#batched-background-migrations) + before you update to a later version [and may take longer than usual](#1400). + ### 13.11.0 Git 2.31.x and later is required. We recommend you use the -- GitLab From 14ac8e92bc935d9cf07dc7feca0d10136070a219 Mon Sep 17 00:00:00 2001 From: Ben Prescott Date: Thu, 7 Oct 2021 11:40:58 +0100 Subject: [PATCH 2/4] Link direct to batched migrations check heading --- doc/user/admin_area/monitoring/background_migrations.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index da245300e1d02d..66001a987a4ba4 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -24,7 +24,7 @@ prevent integer overflow for some tables. ## Check the status of background migrations All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md). -You can [check the status of existing migrations](../../../update/index.md#checking-for-background-migrations-before-upgrading). +You can [check the status of existing migrations](../../../update/index.md#batched-background-migrations). ## Enable or disable batched background migrations -- GitLab From 8346e0ce9b59f377edd63b885252d578e3fc3e7f Mon Sep 17 00:00:00 2001 From: Marcia Ramos Date: Mon, 8 Nov 2021 10:04:04 +0000 Subject: [PATCH 3/4] TW review of 14.x upgrade issues --- doc/update/index.md | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/doc/update/index.md b/doc/update/index.md index 65e0c2fc4ce603..8a24861bef9098 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -89,13 +89,13 @@ Decrease the time required to complete these migrations by increasing the number [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -**For Omnibus installations** +**For Omnibus installations:** ```shell sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' ``` -**For installations from source** +**For installations from source:** ```shell cd /home/git/gitlab @@ -106,10 +106,9 @@ sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMi GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md). -NOTE: Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade. -#### Status check +#### Check the status of batched background migrations To check the status of batched background migrations: @@ -131,13 +130,13 @@ The status of batched background migrations can also be queried directly in the ``` If the migrations are not finished and you try to update to a later version, -you'll see an error like: +GitLab prompts you with an error: ```plaintext Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` -[Read about the options for completing an upgrade if this happens](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). +If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. ### What do I do if my background migrations are stuck? @@ -170,9 +169,9 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` -**Batched migrations (GitLab 14.0 and newer)** +**Batched migrations (GitLab 14.0 and newer):** -[Read more about troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). +See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). ## Dealing with running CI/CD pipelines and jobs @@ -347,13 +346,10 @@ Git 2.33.x and later is required. We recommend you use the ### 14.3.0 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). - - Ensure [batched background migrations finish](#batched-background-migrations) before upgrading to 14.3.Z from earlier GitLab 14 releases. - - Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) for how to proceed. - - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) @@ -375,12 +371,9 @@ for how to proceed. ### 14.2.0 - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). - - Ensure [batched background migrations finish](#batched-background-migrations) before upgrading to 14.2.Z from earlier GitLab 14 releases. - - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) @@ -405,7 +398,6 @@ for how to proceed. - [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) but can upgrade to 14.1.Z. - - It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. 14.1 is included on the upgrade path for the broadest compatibility with self-managed installations, and ensure 14.0.0-14.0.4 installations do not @@ -415,7 +407,6 @@ for how to proceed. - Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher. - - Due to an issue where `BatchedBackgroundMigrationWorkers` were [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) @@ -445,11 +436,9 @@ for how to proceed. - Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, because of [batched background migrations](#batched-background-migrations). - 1. Upgrade first to either: - 14.0.5 or a later 14.0.Z patch release. - 14.1.0 or a later 14.1.Z patch release. - 1. [Batched background migrations need to finish](#batched-background-migrations) before you update to a later version [and may take longer than usual](#1400). -- GitLab From 4806a25e009fc958cd5e4c7b552f2af5a6412ece Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ben=20Prescott=20=40bprescott=5F=E2=86=99=20=E2=98=BA?= Date: Tue, 9 Nov 2021 08:26:27 +0000 Subject: [PATCH 4/4] Bump upgrade path to 13.12.15 --- doc/update/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/update/index.md b/doc/update/index.md index 8a24861bef9098..216ead2128f223 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -222,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.12`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. -- GitLab