From 3a7a89b6b23fabc70fd351fdd13c516e2293849c Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 6 Nov 2019 17:54:48 -0600 Subject: [PATCH 1/8] Add a design document for gitlab.com security development --- README.md | 4 + general/security/future.md | 331 +++++++++++++++++++++++++++++++++++++ 2 files changed, 335 insertions(+) create mode 100644 general/security/future.md diff --git a/README.md b/README.md index dd9a1ddf..36f3c05d 100644 --- a/README.md +++ b/README.md @@ -58,6 +58,10 @@ general process overview and specific documentation for different stakeholders. - [Release template files](https://gitlab.com/gitlab-org/release-tools/tree/master/templates) - [How to create a blog post for a patch release](general/patch/blog-post.md) +## Upcoming changes + +- [Security development on gitlab.com](general/security/future.md) design document + ## Further Reading - ["Release Manager - The invisible hero"](https://about.gitlab.com/2015/06/25/release-manager-the-invisible-hero/) (2015-06-25) diff --git a/general/security/future.md b/general/security/future.md new file mode 100644 index 00000000..ec40fc26 --- /dev/null +++ b/general/security/future.md @@ -0,0 +1,331 @@ +# Security development on gitlab.com + +## Goals + +1. :white_check_mark: Continue to develop new features and bug fixes in public projects on gitlab.com. +1. :exclamation: Develop security fixes in private projects on gitlab.com. +1. :white_check_mark: Continue to automatically deploy new features and bug fixes to gitlab.com for real-world testing. +1. :exclamation: Automatically deploy *undisclosed* security fixes to gitlab.com for up-to-date security. +1. :white_check_mark: Continue to release a new minor version of GitLab every month on the 22nd, built from a specific snapshot of gitlab.com's deployed version. +1. :white_check_mark: Continue to regularly publish patch versions for the latest minor version to resolve bugs. +1. :white_check_mark: Continue to regularly publish security versions for the three most recent minor versions to resolve security vulnerabilities. + +## Definitions + +### Projects + + * **Canonical**: + * **Security**: + * **Build**: + +Throughout this document, when referencing **Canonical**, **Security**, or **Build** +with **Bold Title Case**, we're referencing these projects. + +For the sake of example, we'll be focusing on a single project, GitLab EE, +but this setup expands to all GitLab components that need security fixes +(Shell, Pages, Gitaly, etc.). + +### Branches + + * `master`: the default branch + * `stable`: Any versioned stable branch, e.g., `12-3-stable` + * `auto-deploy`: Any versioned auto-deploy branch, e.g., `12-3-auto-deploy-20190911` + * `security/*`: One of the above branches mirrored to **Security** via pull mirror with branch prefix. + * `security/master-canonical`: A special read-only clone of **Canonical** `master`. + * `security/upstream`: A temporary **Security** branch used to bring published changes back into **Canonical** `master`. + +Throughout this document, when referencing a branch in `backticks`, or as a +graph node, we're referencing these branches. + +## Mirroring + +**Canonical** syncs protected branches to **Build** via push mirroring: + +```mermaid +graph TD + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + subgraph Canonical + c-master(master) + c-stable(stable) + c-feature(feature/some-new-feature) + c-bug(15015-some-bug-fix) + end + + subgraph Build + b-master(master) + b-stable(stable) + end + + c-master -->|push mirror| b-master + c-stable -->|push mirror| b-stable + + class c-feature,c-bug fade + class c-master,c-stable canonical + class b-master,b-stable canonical +``` + +**Security** receives protected branches from **Canonical** via pull mirroring with a `security/` branch prefix: + +```mermaid +graph TD + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + subgraph Security + s-master(security/master) + s-stable(security/stable) + end + + subgraph Canonical + c-master(master) + c-stable(stable) + c-feature(feature/some-new-feature) + c-bug(15015-some-bug-fix) + end + + c-master ---|pull mirror| s-master + c-stable ---|pull mirror| s-stable + + class c-master,c-stable,c-auto-deploy canonical + class s-master,s-stable,s-auto-deploy security + class c-feature,c-bug fade +``` + +> :warning: The prefixed `security/*` branches are targets for **Security** merge requests that resolve security issues, and will regularly diverge from their **Canonical** source. + +For reasons that will be explained later in this document, **Security** needs +a pristine version of **Canonical** `master` that never diverges. This branch +is synced via a scheduled [Merge Train] job: + +[Merge Train]: https://gitlab.com/gitlab-org/merge-train + +```mermaid +graph TD + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + subgraph Canonical + c-master(master) + c-stable(stable) + c-feature(feature/some-new-feature) + c-bug(15015-some-bug-fix) + end + + subgraph Security + s-master(security/master) + s-stable(security/stable) + sc-master(security/master-canonical) + end + + c-master -->|pull mirror| s-master + c-stable -->|pull mirror| s-stable + c-master -.-> |merge-train| sc-master + + class c-master,c-stable,c-auto-deploy,sc-master canonical + class s-master,s-stable,s-auto-deploy security + class c-feature,c-bug fade +``` + +**Security** syncs protected branches to **Build** via push mirroring: + +```mermaid +graph TD + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + subgraph Security + s-master(security/master) + s-stable(security/stable) + end + + subgraph Build + bs-master(security/master) + bs-stable(security/stable) + + b-master(master) + b-stable(stable) + end + + s-master -->|push mirror| bs-master + s-stable -->|push mirror| bs-stable + + class b-master,b-stable,b-auto-deploy canonical + class s-master,s-stable,s-auto-deploy security + class bs-master,bs-stable,bs-auto-deploy security + class b-master,b-stable fade +``` + +## Normal development + +Nothing changes for day-to-day development, so we will not document that +procedure in detail here. + +## Security development + +Developers work on security fixes in **Security**, targeting `security/master`. A +fix goes through review, gets merged, and then gets the `unpublished` label. +The purpose of this label is described later. + +Once a fix is merged into `security/master`, a developer can open backport +MRs against the `security/stable` branches. + +```mermaid +graph TD + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + subgraph Security + s-master(security/master) + + s-stable(security/stable) + s-auto-deploy(security/auto-deploy) + s-developer(security/fix-mermaid-xss) + s-developer-backport(security/fix-mermaid-xss-backport) + end + + subgraph Build + b-master(master) + b-stable(stable) + + bs-master(security/master) + bs-stable(security/stable) + bs-auto-deploy(security/auto-deploy) + end + + s-developer -->|merge| s-master + s-master -.->|cherry-pick| s-auto-deploy + s-developer-backport -->|merge| s-stable + + s-master -->|push mirror| bs-master + s-stable -->|push mirror| bs-stable + s-auto-deploy -->|push mirror| bs-auto-deploy + + class sc-master,sc-stable canonical + class s-master,s-stable,s-auto-deploy security + class b-master,b-stable,b-auto-deploy canonical + class bs-master,bs-stable,bs-auto-deploy security + class s-developer,s-developer-backport fade +``` + +## Auto-deploys + +See the [existing documentation](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/auto-deploy.md), +with two key differences: + +1. The auto-deploy branch is created in **Security** with a `security/` prefix: + + ```mermaid + sequenceDiagram + participant RT as release-tools + participant C as Canonical + participant S as Security + + note left of RT: Weekly + + rect rgb(116, 189, 61) + RT->>C:Latest green `master` + C->>RT:d1f38742 + end + + rect rgb(255, 114, 114) + RT->>S:Create `security/auto-deploy` from d1f38742 + end + ``` + +2. We cherry-pick merge requests labeled `Pick into auto-deploy` from **Canonical** _and_ **Security**. + + ```mermaid + sequenceDiagram + participant RT as release-tools + participant C as Canonical + participant S as Security + + rect rgb(116, 189, 61) + RT->>C:MRs with `Pick into auto-deploy` + C->>RT:!5483 + RT->>C:Merge commit SHA for !5483 + C->>RT:601cc959 + RT->>S:Cherry-pick 601cc959 into `security/auto-deploy` + end + + rect rgb(255, 114, 114) + RT->>S:MRs with `Pick into auto-deploy` + S->>RT:!549 + RT->>S:Merge commit SHA for !549 + S->>RT:2a150e3e + RT->>S:Cherry-pick 2a150e3e into `security/auto-deploy` + end + ``` + +This ensures gitlab.com is running a version of the codebase that has a +passing build with the latest bug fixes and features, as well as all known +security fixes. + +## Monthly releases + +Same as before, but using a **Security** `security/auto-deploy` branch as the +source. + +## Patch releases + +Same as before. + +## Security patch releases + +Same as before, but using **Security**'s `security/stable` branches as the +source. + +## Branch syncing after publication + +After a security fix is published, we need to get it from **Security**'s +`security/master` into **Canonical**'s `master`. However, `security/master` +may contain fixes that haven't yet been published, so we can't simply merge +the two branches. + +This is where the `unpublished` label comes into play. When we publish a +security release, we: + +1. Create a `security/upstream` branch in **Security** based off of `security/master-canonical`. +1. Gather `unpublished` **Security** merge requests. +1. Cherry-pick the merge commits into `security/upstream`. +1. Open an upstream merge request in **Canonical** with a source of `security/upstream` and a target of `master`. +1. Automatically Approve and "Merge when pipeline succeeds" the merge request. + +Additionally, the `security/stable` branches on **Build** get merged into +their non-prefixed counterparts and pushed to **Canonical**: + +```mermaid +sequenceDiagram + participant B as Build + participant RT as release-tools + participant C as Canonical + + B->>RT:pull `stable` + B->>RT:pull `security/stable` + RT-->>RT:merge `security/stable` into `stable` + + RT->>C:push `stable` + C->>B:Mirror `stable` +``` + +Here's an overview of the *post-publication* state of all projects and branches: + +* **Canonical** + * `master`: Receiving active development, contains all _released_ security fixes. + * `stable`: Contains all released security backports. +* **Security**: + * `security/master`: Contains latest changes from **Canonical** `master`, plus unreleased security fixes. + * `security/stable`: Contains all released security backports. + * `security/auto-deploy`: Contains latest changes from a green **Canonical** `master`, plus unreleased security fixes. +* **Build**: All protected branches are synced from **Canonical** and **Security**. + +--- + +[Return to Guides](../README.md) -- GitLab From 89afcf4d84d17497d18211480ca77c288a1642be Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 6 Nov 2019 18:24:08 -0600 Subject: [PATCH 2/8] Add a mirroring overview --- general/security/future.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/general/security/future.md b/general/security/future.md index ec40fc26..ec67c3e4 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -39,6 +39,34 @@ graph node, we're referencing these branches. ## Mirroring +Since there's a lot happening between multiple projects, it may be helpful to +start with a high-level overview: + +```mermaid +graph LR + classDef canonical fill:#74bd3d,stroke:#333,stroke-width:1px + classDef security fill:#ff7272,stroke:#333,stroke-width:1px + classDef fade opacity:0.8,stroke-dasharray:4 4 + + canonical((Canonical)) + security((Security)) + build((Build)) + rt{release-tools} + + canonical -->|push mirror| build + canonical -->|pull mirror| security + security -->|push mirror| build + + build -.-> |git clone| rt + rt -.-> |rake publish| canonical + + class security security + class canonical canonical + style build fill:#72b9ff +``` + +Now let's break it down into smaller, more detailed parts. + **Canonical** syncs protected branches to **Build** via push mirroring: ```mermaid -- GitLab From c100687b360355cc1688574d7e4ee8f9bd213444 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Wed, 6 Nov 2019 18:29:41 -0600 Subject: [PATCH 3/8] Resolve style linting --- general/security/future.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/general/security/future.md b/general/security/future.md index ec67c3e4..468240c8 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -14,9 +14,9 @@ ### Projects - * **Canonical**: - * **Security**: - * **Build**: +* **Canonical**: +* **Security**: +* **Build**: Throughout this document, when referencing **Canonical**, **Security**, or **Build** with **Bold Title Case**, we're referencing these projects. @@ -27,12 +27,12 @@ but this setup expands to all GitLab components that need security fixes ### Branches - * `master`: the default branch - * `stable`: Any versioned stable branch, e.g., `12-3-stable` - * `auto-deploy`: Any versioned auto-deploy branch, e.g., `12-3-auto-deploy-20190911` - * `security/*`: One of the above branches mirrored to **Security** via pull mirror with branch prefix. - * `security/master-canonical`: A special read-only clone of **Canonical** `master`. - * `security/upstream`: A temporary **Security** branch used to bring published changes back into **Canonical** `master`. +* `master`: the default branch +* `stable`: Any versioned stable branch, e.g., `12-3-stable` +* `auto-deploy`: Any versioned auto-deploy branch, e.g., `12-3-auto-deploy-20190911` +* `security/*`: One of the above branches mirrored to **Security** via pull mirror with branch prefix. +* `security/master-canonical`: A special read-only clone of **Canonical** `master`. +* `security/upstream`: A temporary **Security** branch used to bring published changes back into **Canonical** `master`. Throughout this document, when referencing a branch in `backticks`, or as a graph node, we're referencing these branches. -- GitLab From d2183a1ad6c0e6831d421cd8ac5381b79c280d0d Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 7 Nov 2019 16:00:13 -0600 Subject: [PATCH 4/8] Add arrows to pull mirror --- general/security/future.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/general/security/future.md b/general/security/future.md index 468240c8..db82c7b9 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -115,8 +115,8 @@ graph TD c-bug(15015-some-bug-fix) end - c-master ---|pull mirror| s-master - c-stable ---|pull mirror| s-stable + c-master -->|pull mirror| s-master + c-stable -->|pull mirror| s-stable class c-master,c-stable,c-auto-deploy canonical class s-master,s-stable,s-auto-deploy security -- GitLab From 2e8e5173cbe7478755ecd24e72e52fdfca50d91b Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 7 Nov 2019 16:00:55 -0600 Subject: [PATCH 5/8] Shout out Omnibus --- general/security/future.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/general/security/future.md b/general/security/future.md index db82c7b9..f74f47e2 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -23,7 +23,7 @@ with **Bold Title Case**, we're referencing these projects. For the sake of example, we'll be focusing on a single project, GitLab EE, but this setup expands to all GitLab components that need security fixes -(Shell, Pages, Gitaly, etc.). +(Omnibus, Shell, Pages, Gitaly, etc.). ### Branches -- GitLab From 7096b390c480d2c8c0cd68590cf8e45ea4985ad9 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 7 Nov 2019 16:01:18 -0600 Subject: [PATCH 6/8] Remove Emoji from Goals --- general/security/future.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/general/security/future.md b/general/security/future.md index f74f47e2..b8ff8bc8 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -2,13 +2,13 @@ ## Goals -1. :white_check_mark: Continue to develop new features and bug fixes in public projects on gitlab.com. -1. :exclamation: Develop security fixes in private projects on gitlab.com. -1. :white_check_mark: Continue to automatically deploy new features and bug fixes to gitlab.com for real-world testing. -1. :exclamation: Automatically deploy *undisclosed* security fixes to gitlab.com for up-to-date security. -1. :white_check_mark: Continue to release a new minor version of GitLab every month on the 22nd, built from a specific snapshot of gitlab.com's deployed version. -1. :white_check_mark: Continue to regularly publish patch versions for the latest minor version to resolve bugs. -1. :white_check_mark: Continue to regularly publish security versions for the three most recent minor versions to resolve security vulnerabilities. +1. Continue to develop new features and bug fixes in public projects on gitlab.com. +1. Develop security fixes in private projects on gitlab.com. +1. Continue to automatically deploy new features and bug fixes to gitlab.com for real-world testing. +1. Automatically deploy *undisclosed* security fixes to gitlab.com for up-to-date security. +1. Continue to release a new minor version of GitLab every month on the 22nd, built from a specific snapshot of gitlab.com's deployed version. +1. Continue to regularly publish patch versions for the latest minor version to resolve bugs. +1. Continue to regularly publish security versions for the three most recent minor versions to resolve security vulnerabilities. ## Definitions -- GitLab From d33985f2f9460574fc6661f6a62c9e29e6fcbb2a Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 7 Nov 2019 16:02:42 -0600 Subject: [PATCH 7/8] Reword Definitions --- general/security/future.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/general/security/future.md b/general/security/future.md index b8ff8bc8..cd448c08 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -14,6 +14,12 @@ ### Projects +For the sake of example, we'll be focusing on a single project, GitLab EE, +but this setup expands to all GitLab components that need security fixes +(Omnibus, Shell, Pages, Gitaly, etc.). + +The project has three locations: **Canonical**, **Security**, and **Build**. + * **Canonical**: * **Security**: * **Build**: @@ -21,10 +27,6 @@ Throughout this document, when referencing **Canonical**, **Security**, or **Build** with **Bold Title Case**, we're referencing these projects. -For the sake of example, we'll be focusing on a single project, GitLab EE, -but this setup expands to all GitLab components that need security fixes -(Omnibus, Shell, Pages, Gitaly, etc.). - ### Branches * `master`: the default branch -- GitLab From 4e1c6f7dbfb0e5021adb40ee1366229933162773 Mon Sep 17 00:00:00 2001 From: Robert Speicher Date: Thu, 7 Nov 2019 16:07:30 -0600 Subject: [PATCH 8/8] Reword Branches section --- general/security/future.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/general/security/future.md b/general/security/future.md index cd448c08..bb0f2cf8 100644 --- a/general/security/future.md +++ b/general/security/future.md @@ -31,14 +31,15 @@ with **Bold Title Case**, we're referencing these projects. * `master`: the default branch * `stable`: Any versioned stable branch, e.g., `12-3-stable` -* `auto-deploy`: Any versioned auto-deploy branch, e.g., `12-3-auto-deploy-20190911` -* `security/*`: One of the above branches mirrored to **Security** via pull mirror with branch prefix. -* `security/master-canonical`: A special read-only clone of **Canonical** `master`. -* `security/upstream`: A temporary **Security** branch used to bring published changes back into **Canonical** `master`. +* `security/master` and `security/stable`: One of the above branches mirrored to **Security** via pull mirror with branch prefix. +* `security/auto-deploy`: Any versioned auto-deploy branch, e.g., `security/12-3-auto-deploy-20190911` Throughout this document, when referencing a branch in `backticks`, or as a graph node, we're referencing these branches. +There are two special branches, `security/upstream` and +`security/master-canonical` that are detailed in later sections. + ## Mirroring Since there's a lot happening between multiple projects, it may be helpful to -- GitLab