diff --git a/README.md b/README.md index 403d377f7c23e493a52da2640324255f83aeea05..3c0f662bafce83ea603ed5f64e528de302bcb948 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,7 @@ general process overview and specific documentation for different stakeholders. ## Guides +- [Hot patching production](runbooks/hot-patching-production.md) - [Overview of our Tooling](general/tooling.md) - [PDM - Post deploy migrations](https://gitlab.com/gitlab-org/release/docs/-/tree/master/general/post_deploy_migration) - [Overview of Building Packages](general/building-packages.md) diff --git a/runbooks/hot-patching-production.md b/runbooks/hot-patching-production.md new file mode 100644 index 0000000000000000000000000000000000000000..1e2ae9ef9765d1cc769e5ca57d6e41b435c929d5 --- /dev/null +++ b/runbooks/hot-patching-production.md @@ -0,0 +1,21 @@ +# Hot patching production + +This is a runbook to hotpatch production in case of high-severity incidents that can't be mitigated by deplyoing a new package or rollback to a previous package. + +1. Initiate patcher using `/chatops run hotpatch --incident ` +2. Engage the SRE on call/RM that you are going to hotpatch production +3. Drain canary using: `/chatops run canary --disable --production` +4. Lock canary to prevent deployments: `/chatops run deploy lock gprd-cny` +5. `Patcher` gonna open an MR with patch directories reflected with the same `SHA` that production uses. _There could be another directory for the `SHA` on gprd-cny or gstg-cny._ +6. inside `Patcher` repository fetch this MR's branch locally using: `git fetch origin BRANCH_NAME` +7. Create a working branch in [gitlab](https://gitlab.com/gitlab-org/gitlab-ee/) from the current SHA running on production (run `/chatops run auto_deploy status` to find out what this is or see the version from the output of Chatops hotpatch command as described in (1.). + * For example, you can do `git checkout -b patch/my-fix 0bf60009bec` if you want + to derive your branch from `0bf60009bec` revision based on the sha. +8. Make your code changes + * You can cherry-pick commits. +9. Run the command `git --no-pager diff --color=never 0bf60009bec.. -- . ':!spec' ':!ee/spec' > path/to/patch.patch` + * **Note**: this is an example - if you have changed non-spec files in other directories, be sure to include those +10. Copy the patch file to corresponding placeholder directory for the environment that is being patched on the branch created by the Chatops hotpatch command earlier. + Example, to patch release `12.6.201912031517-0bf60009bec.8dfcd02384a` copy the patch file(s) into https://ops.gitlab.net/gitlab-com/engineering/patcher/tree/master/patches/12.6.201912031517-0bf60009bec.8dfcd02384a +11. After creating the MR inform the RM/SRE-on-call that the patch is ready and the MR needs to be approved and merged. +12. Once the MR is merged, an MR will be created in https://ops.gitlab.net/gitlab-com/gl-infra/patcher/-/pipelines. Keep an eye on the pipeline and make sure it passes.