From d5258569201870eab0e1355729762d26f7b4089f Mon Sep 17 00:00:00 2001 From: Ahmad Tolba Date: Mon, 21 Aug 2023 15:10:31 +0200 Subject: [PATCH 1/7] Add hotpatcher runbook --- README.md | 1 + runbooks/hot-patching-production.md | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) create mode 100644 runbooks/hot-patching-production.md diff --git a/README.md b/README.md index 403d377f..3c0f662b 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,7 @@ general process overview and specific documentation for different stakeholders. ## Guides +- [Hot patching production](runbooks/hot-patching-production.md) - [Overview of our Tooling](general/tooling.md) - [PDM - Post deploy migrations](https://gitlab.com/gitlab-org/release/docs/-/tree/master/general/post_deploy_migration) - [Overview of Building Packages](general/building-packages.md) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md new file mode 100644 index 00000000..20d32ec1 --- /dev/null +++ b/runbooks/hot-patching-production.md @@ -0,0 +1,21 @@ +# Hot patching production + +This is a quick runbook to quickly hotpatch production in case of incidents. + +- Initiate patcher using `/chatops run hotpatch --incident ` +- Engage the SRE on call/RM that you are going to hotpatch production +- Drain canary using: `/chatops run canary --disable --production` +- Lock canary to prevent deployments: `/chatops run deploy lock gprd-cny` +- `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production/staging use +- inside `Patcher` repository Fetch this MR's branch locally using: `git fetch origin BRANCH_NAME` +- Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as describe in (4.). + * For example, you can do `git checkout -b patch/my-fix 0bf60009bec` if you want + to derive your branch from `0bf60009bec` revision based on the sha. +- Make your code changes + * You can cherry-pick commits. +- Run the command `git --no-pager diff --color=never 0bf60009bec.. -- . ':!spec' ':!ee/spec' > path/to/patch.patch` + * **Note**: this is an example - if you have changed non-spec files in other directories, be sure to include those +- Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. + Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a +- After creating the MR inform RM/SRE-on-call that the patch is ready and gonna be applied + -- GitLab From b2669975522bdaf24503edf621282e4067569e0b Mon Sep 17 00:00:00 2001 From: Ahmad Tolba Date: Mon, 21 Aug 2023 14:56:10 +0000 Subject: [PATCH 2/7] Update hot-patching-production.md --- runbooks/hot-patching-production.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index 20d32ec1..00c1d749 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -18,4 +18,6 @@ This is a quick runbook to quickly hotpatch production in case of incidents. - Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a - After creating the MR inform RM/SRE-on-call that the patch is ready and gonna be applied - +- After merging and testing patcher remember to: + - Enable canary using: `/chatops run canary --enable --production` + - Unlock canary to prevent deployments: `/chatops run deploy unlock gprd-cny` -- GitLab From 3c7df3752597ee795a9f3d98a5a7f72dee5cb8c1 Mon Sep 17 00:00:00 2001 From: Ahmad Tolba Date: Mon, 21 Aug 2023 14:56:41 +0000 Subject: [PATCH 3/7] Update hot-patching-production.md --- runbooks/hot-patching-production.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index 00c1d749..ada93a96 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -19,5 +19,5 @@ This is a quick runbook to quickly hotpatch production in case of incidents. Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a - After creating the MR inform RM/SRE-on-call that the patch is ready and gonna be applied - After merging and testing patcher remember to: - - Enable canary using: `/chatops run canary --enable --production` - - Unlock canary to prevent deployments: `/chatops run deploy unlock gprd-cny` + - Enable canary traffic using: `/chatops run canary --enable --production` + - Unlock canary to enable new deployments: `/chatops run deploy unlock gprd-cny` -- GitLab From 34f1af7dce69c58ac799f66fcfd054f755923f8f Mon Sep 17 00:00:00 2001 From: Ahmad Tolba Date: Tue, 22 Aug 2023 10:28:12 +0200 Subject: [PATCH 4/7] Address MR comments --- runbooks/hot-patching-production.md | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index ada93a96..b8136841 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -2,22 +2,23 @@ This is a quick runbook to quickly hotpatch production in case of incidents. -- Initiate patcher using `/chatops run hotpatch --incident ` -- Engage the SRE on call/RM that you are going to hotpatch production -- Drain canary using: `/chatops run canary --disable --production` -- Lock canary to prevent deployments: `/chatops run deploy lock gprd-cny` -- `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production/staging use -- inside `Patcher` repository Fetch this MR's branch locally using: `git fetch origin BRANCH_NAME` -- Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as describe in (4.). +1. Initiate patcher using `/chatops run hotpatch --incident ` +2. Engage the SRE on call/RM that you are going to hotpatch production +3. Drain canary using: `/chatops run canary --disable --production` +4. Lock canary to prevent deployments: `/chatops run deploy lock gprd-cny` +5. `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production uses _there could be another directory for gprd-cny or gstg-cny_ +6. inside `Patcher` repository fetch this MR's branch locally using: `git fetch origin BRANCH_NAME` +7. Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as describe in (1.). * For example, you can do `git checkout -b patch/my-fix 0bf60009bec` if you want to derive your branch from `0bf60009bec` revision based on the sha. -- Make your code changes +8. Make your code changes * You can cherry-pick commits. -- Run the command `git --no-pager diff --color=never 0bf60009bec.. -- . ':!spec' ':!ee/spec' > path/to/patch.patch` +9. Run the command `git --no-pager diff --color=never 0bf60009bec.. -- . ':!spec' ':!ee/spec' > path/to/patch.patch` * **Note**: this is an example - if you have changed non-spec files in other directories, be sure to include those -- Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. +10. Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a -- After creating the MR inform RM/SRE-on-call that the patch is ready and gonna be applied -- After merging and testing patcher remember to: +11. After creating the MR inform the RM/SRE-on-call that the patch is ready and the MR needs to be approved and merged. +12. The pipeline could be found here: https://ops.gitlab.net/gitlab-com/gl-infra/patcher/-/pipelines +13. After merging and testing patcher remember to: - Enable canary traffic using: `/chatops run canary --enable --production` - Unlock canary to enable new deployments: `/chatops run deploy unlock gprd-cny` -- GitLab From b246135692d195c48362a0b5ae549902854ef337 Mon Sep 17 00:00:00 2001 From: Reuben Pereira <2967854-rpereira2@users.noreply.gitlab.com> Date: Tue, 22 Aug 2023 08:46:50 +0000 Subject: [PATCH 5/7] Apply 4 suggestion(s) to 1 file(s) --- runbooks/hot-patching-production.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index b8136841..6de3e24b 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -6,9 +6,9 @@ This is a quick runbook to quickly hotpatch production in case of incidents. 2. Engage the SRE on call/RM that you are going to hotpatch production 3. Drain canary using: `/chatops run canary --disable --production` 4. Lock canary to prevent deployments: `/chatops run deploy lock gprd-cny` -5. `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production uses _there could be another directory for gprd-cny or gstg-cny_ +5. `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production uses. _There could be another directory for the `SHA` on gprd-cny or gstg-cny._ 6. inside `Patcher` repository fetch this MR's branch locally using: `git fetch origin BRANCH_NAME` -7. Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as describe in (1.). +7. Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as described in (1.). * For example, you can do `git checkout -b patch/my-fix 0bf60009bec` if you want to derive your branch from `0bf60009bec` revision based on the sha. 8. Make your code changes @@ -18,7 +18,7 @@ This is a quick runbook to quickly hotpatch production in case of incidents. 10. Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a 11. After creating the MR inform the RM/SRE-on-call that the patch is ready and the MR needs to be approved and merged. -12. The pipeline could be found here: https://ops.gitlab.net/gitlab-com/gl-infra/patcher/-/pipelines -13. After merging and testing patcher remember to: +12. Once the MR is merged, an MR will be created in https://ops.gitlab.net/gitlab-com/gl-infra/patcher/-/pipelines. Keep an eye on the pipeline and make sure it passes. +13. After the Patcher pipeline completes, remember to: - Enable canary traffic using: `/chatops run canary --enable --production` - Unlock canary to enable new deployments: `/chatops run deploy unlock gprd-cny` -- GitLab From 2ab9924a409465deefb36ccf26e5d8063b5ca6bc Mon Sep 17 00:00:00 2001 From: Ahmad Tolba Date: Tue, 22 Aug 2023 10:49:48 +0200 Subject: [PATCH 6/7] Address MR comments --- runbooks/hot-patching-production.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index 6de3e24b..7ff2f6d2 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -19,6 +19,3 @@ This is a quick runbook to quickly hotpatch production in case of incidents. Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a 11. After creating the MR inform the RM/SRE-on-call that the patch is ready and the MR needs to be approved and merged. 12. Once the MR is merged, an MR will be created in https://ops.gitlab.net/gitlab-com/gl-infra/patcher/-/pipelines. Keep an eye on the pipeline and make sure it passes. -13. After the Patcher pipeline completes, remember to: - - Enable canary traffic using: `/chatops run canary --enable --production` - - Unlock canary to enable new deployments: `/chatops run deploy unlock gprd-cny` -- GitLab From 50aa4f7ad0a92614628e14c4e6ba7467526a873a Mon Sep 17 00:00:00 2001 From: Michele Bursi Date: Tue, 22 Aug 2023 10:27:11 +0000 Subject: [PATCH 7/7] Apply 1 suggestion(s) to 1 file(s) --- runbooks/hot-patching-production.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md index 7ff2f6d2..1e2ae9ef 100644 --- a/runbooks/hot-patching-production.md +++ b/runbooks/hot-patching-production.md @@ -1,6 +1,6 @@ # Hot patching production -This is a quick runbook to quickly hotpatch production in case of incidents. +This is a runbook to hotpatch production in case of high-severity incidents that can't be mitigated by deplyoing a new package or rollback to a previous package. 1. Initiate patcher using `/chatops run hotpatch --incident ` 2. Engage the SRE on call/RM that you are going to hotpatch production -- GitLab