diff --git a/release_manager/release_supervisor.md b/release_manager/release_supervisor.md deleted file mode 100644 index acb0152afdcd3428654d755f45266042cddf9278..0000000000000000000000000000000000000000 --- a/release_manager/release_supervisor.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -tags: - - auto-deploy - - release-management ---- -# Release Supervisor - -The release supervisor role was introduced in September 2025 and mirrors the Release Manager role. Release supervisors drive the monthly release of GitLab and the daily deployments to GitLab.com (autodeploy process). - -## Responsibilities - -You as a release supervisor have a responsibility to help deliver the work of every -single person involved in creating the product and running the application at -GitLab.com. - -## General Release process overview - -[GitLab.com deployments](https://handbook.gitlab.com/handbook/engineering/releases/#gitlabcom-deployments) - -### Daily -Help with the [auto-deploy process](../general/deploy/overview-of-the-auto-deploy-pipelines.md) for GitLab.com. - -1. Keep an eye on #f_upcoming_release - most daily activities and handoffs are done here. -2. Watch and improve the [auto-build pipelines](../general/auto-build.md) -3. Promoting packages to [deploy to GitLab.com](../general/deploy/gitlab-com-deployer.md) - to be updated -4. Handling [deployment failures](../general/deploy/failures.md) -5. Running Post Deploy Migrations at appropriate times- [PDM](../general/database-migrations/post-deploy-migration/readme.md) - -Areas we are watching and looking to make improvements: -1. Slow jobs - what can be sped up. -2. Commonly erroring jobs - what can be made more reliable. -3. Automate existing deployments and release process as much as posible - -Ideally, you will have Developer roles to projects with affected pipelines so you can see and MR any code / tooling that could be improved. If you can't see something, talk to a Release Manager and we'll work to get you access. - -### Weekly (2nd/3rd week of the month) -Help with the [monthly release](https://handbook.gitlab.com/handbook/engineering/releases/monthly-releases/). - -The monthly release process is [here](../general/monthly/process.md). We will likely start you with shadowing this and then update our templates to more clearly assign tasks and todos. Sample [monthly release issue from 18.5](https://gitlab.com/gitlab-org/release/tasks/-/issues/21396). - -## Getting help from the Release Manager - -There are a few ways to find the Release Managers (RM). First, the schedule is posted at https://about.gitlab.com/community/release-managers/. - -Most daily discussion/handoffs and priorities are discussed in #f_upcoming_release. - -To get ahold of an RM, use the process for [How to reach Release management](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#reaching-our-team) process. Very quickly - use `@release-managers` in Slack to get ahold of the group. - -## Glossary -We have a glossary [here](../general/deploy/glossary.md) to help with what things mean. Feel free to MR and add to it. diff --git a/release_manager/release_supervisor/.gitkeep b/release_manager/release_supervisor/.gitkeep new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/release_manager/release_supervisor/commit_diff_between_versions.png b/release_manager/release_supervisor/commit_diff_between_versions.png new file mode 100644 index 0000000000000000000000000000000000000000..9a3cf8b7b3cc016dfc40e9c00c3a75f3848a552e Binary files /dev/null and b/release_manager/release_supervisor/commit_diff_between_versions.png differ diff --git a/release_manager/release_supervisor/gprd_finished_slack.png b/release_manager/release_supervisor/gprd_finished_slack.png new file mode 100644 index 0000000000000000000000000000000000000000..de01920f2f2f05360ce4528154245c905a6d5e44 Binary files /dev/null and b/release_manager/release_supervisor/gprd_finished_slack.png differ diff --git a/release_manager/release_supervisor/is_pdm_necessary.png b/release_manager/release_supervisor/is_pdm_necessary.png new file mode 100644 index 0000000000000000000000000000000000000000..9026baae8064fa6c03afc28e0988bc460e961ca0 Binary files /dev/null and b/release_manager/release_supervisor/is_pdm_necessary.png differ diff --git a/release_manager/release_supervisor/no_release_blockers.png b/release_manager/release_supervisor/no_release_blockers.png new file mode 100644 index 0000000000000000000000000000000000000000..1dfc789a2d835717b6ce51936802a3fa6eeb5332 Binary files /dev/null and b/release_manager/release_supervisor/no_release_blockers.png differ diff --git a/release_manager/release_supervisor/production_checks_passed_baking_complete.png b/release_manager/release_supervisor/production_checks_passed_baking_complete.png new file mode 100644 index 0000000000000000000000000000000000000000..1aaae9183eec9a425b94201ce2c2b2b0b946de0c Binary files /dev/null and b/release_manager/release_supervisor/production_checks_passed_baking_complete.png differ diff --git a/release_manager/release_supervisor/release_supervisor.md b/release_manager/release_supervisor/release_supervisor.md new file mode 100644 index 0000000000000000000000000000000000000000..01ff0cb01af18459e83da6d8ff4a97bd42a2a5e0 --- /dev/null +++ b/release_manager/release_supervisor/release_supervisor.md @@ -0,0 +1,136 @@ +--- +tags: + - auto-deploy + - release-management +--- +# Release Supervisor + +The release supervisor role was introduced in September 2025 and mirrors the Release Manager role. Release supervisors drive the monthly release of GitLab and the daily deployments to GitLab.com (autodeploy process). + +## Responsibilities + +You as a release supervisor have a responsibility to help deliver the work of every +single person involved in creating the product and running the application at +GitLab.com. + +## General Release process overview + +[GitLab.com deployments](https://handbook.gitlab.com/handbook/engineering/releases/#gitlabcom-deployments) + +### Daily +Help with the [auto-deploy process](../general/deploy/overview-of-the-auto-deploy-pipelines.md) for GitLab.com. + +1. Keep an eye on #f_upcoming_release - most daily activities and handoffs are done here. +2. Watch and improve the [auto-build pipelines](../general/auto-build.md) +3. Promoting packages to [deploy to GitLab.com](../general/deploy/gitlab-com-deployer.md) - to be updated +4. Handling [deployment failures](../general/deploy/failures.md) +5. Running Post Deploy Migrations at appropriate times- [PDM](../general/database-migrations/post-deploy-migration/readme.md) + +Areas we are watching and looking to make improvements: +1. Slow jobs - what can be sped up. +2. Commonly erroring jobs - what can be made more reliable. +3. Automate existing deployments and release process as much as posible + +Ideally, you will have Developer roles to projects with affected pipelines so you can see and MR any code / tooling that could be improved. If you can't see something, talk to a Release Manager and we'll work to get you access. + +### Common tasks + +#### Inspect and restart failing jobs, especially QA jobs and pipeline + +Monitor the [#f_upcoming_release](https://gitlab.enterprise.slack.com/archives/C0139MAV672) and [#announcements](https://gitlab.enterprise.slack.com/archives/C8PKBH3M5) Slack channels for notifications about failing pipelines or jobs. When you identify a failure: + +1. **Investigate** - Review the failure details to understand the root cause +2. **Retry** - If appropriate, retry the failing pipeline or job using the chatops command below +3. **Escalate** - If the retry fails, escalate to the Release Managers or the responsible team + +Use this chatops command to retry: +``` +/chatops run ci retry +``` + +When retrying, use the 🔄 symbol to keep the team informed that the pipeline/job restart is currently taking place. Once the retry is successful, mark it with the ✅ symbol. If the retry still fails, add a ❌ symbol and escalate if necessary. + +#### Promote package versions + +Promote a package version once it has completed testing in gstg-cny and gprd-cny, and passed the baking stage. This deploys the package to both gstg and gprd environments. + +**Order of actions:** + +1. Verify that Production checks have passed in [#f_upcoming_release](https://gitlab.enterprise.slack.com/archives/C0139MAV672) + +Production checks passed and baking complete + +2. To make sure that a promotion is possible, run the chatops command in [#f_upcoming_release](https://gitlab.enterprise.slack.com/archives/C0139MAV672): +``` +/chatops run auto_deploy blockers +``` + +No release blockers + +Then look for the gprd finished message in the [#announcements](https://gitlab.enterprise.slack.com/archives/C8PKBH3M5) channel. + +GPRD finished deployment + +3. Check the commit diff between the promotion candidate version (e.g., 18.7.202512122306-8f4c60c9c98.a871df033dd) and what is currently deployed in gprd using [Auto-deploy-view](https://auto-deploy-view-e121e0.gitlab.io/). Target approximately 100 commits per promotion. Fewer commits underutilize the ~2-hour deployment window, while more commits increase the risk of introducing bugs or breaking changes. If the diff exceeds 150 commits and time permits, split the promotion into two packages of approximately 75 commits each. Also check for any other upcoming versions that are about to be ready. Is the next version more than 2 hours away? Will a subsequent version be ready promptly? + +For a more comprehensive commit pressure overview, see the [Release Management dashboard](https://dashboards.gitlab.net/d/delivery-release_management/delivery3a-release-management?from=now-12h&orgId=1&timezone=utc&to=now&var-PROMETHEUS_DS=mimir-gitlab-ops). + +Commit diff between versions + +4. Run the chatops command in [#f_upcoming_release](https://gitlab.enterprise.slack.com/archives/C0139MAV672): +``` +/chatops run auto_deploy promote 18.7.202512122306-8f4c60c9c98.a871df033dd +``` + +Alternatively, you can use the short version format: +``` +/chatops run auto_deploy promote 18.7.202512122306 +``` + +5. You will receive a confirmation message from the chatops command. + +#### Execute post-deploy migrations + +Post-deploy migrations (PDM) should be executed between approximately 06:00 UTC and 09:00 UTC, during the transition between the end of the APAC shift and the beginning of the EMEA shift. + +**Prerequisites:** + +1. Check the [Release Management dashboard](https://dashboards.gitlab.net/d/delivery-release_management/delivery3a-release-management?from=now-12h&orgId=1&timezone=utc&to=now&var-PROMETHEUS_DS=mimir-gitlab-ops) to verify that pending post-deploy migrations exist (count > 0). If there are no pending PDMs, there is no need to execute. + +Is PDM necessary + +2. Verify there are no ongoing incidents by checking the [#incident_management](https://gitlab.enterprise.slack.com/archives/C8PKBH3M5) channel. + +3. Ensure there are no ongoing deployments to gstg or gprd (canary deployments are allowed). Check the [Auto-deploy-view](https://auto-deploy-view-e121e0.gitlab.io/) and running pipelines. + +4. Run the following chatops command in [#f_upcoming_release](https://gitlab.enterprise.slack.com/archives/C0139MAV672) to verify there are no blockers: +``` +/chatops run auto_deploy blockers +``` + +5. Verify that at least 1 hour has passed since the latest deployment to gprd finished. + +GPRD finished deployment + +6. Execute the post-deploy migrations: +``` +/chatops run post_deploy_migrations execute +``` + +**Important:** Once a PDM has been executed, the packages available for rollback will be severely limited. Only packages deployed to gprd during the PDM execution and newer versions can be rolled back to. All previous versions will not be available for rollback. See the [rollback guide](../../runbooks/rollback-a-deployment.md) for more details. + +### Weekly (2nd/3rd week of the month) +Help with the [monthly release](https://handbook.gitlab.com/handbook/engineering/releases/monthly-releases/). + +The monthly release process is [here](../general/monthly/process.md). We will likely start you with shadowing this and then update our templates to more clearly assign tasks and todos. Sample [monthly release issue from 18.5](https://gitlab.com/gitlab-org/release/tasks/-/issues/21396). + +## Getting help from the Release Manager + +There are a few ways to find the Release Managers (RM). First, the schedule is posted at https://about.gitlab.com/community/release-managers/. + +Most daily discussion/handoffs and priorities are discussed in #f_upcoming_release. + +To get ahold of an RM, use the process for [How to reach Release management](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#reaching-our-team) process. Very quickly - use `@release-managers` in Slack to get ahold of the group. + +## Glossary +We have a glossary [here](../general/deploy/glossary.md) to help with what things mean. Feel free to MR and add to it.