From f0dd6b8784d3678b87878000d35e9b8c75ef2444 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Fri, 29 Jul 2022 08:33:06 +0200 Subject: [PATCH 01/17] Draft of blueprint for CI components catalog --- .../ci_pipeline_components/index.md | 213 ++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 doc/architecture/blueprints/ci_pipeline_components/index.md diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md new file mode 100644 index 00000000000000..529a55e6678a47 --- /dev/null +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -0,0 +1,213 @@ +--- +stage: Stage +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +comments: false +description: 'Create a catalog of shareable pipeline constructs' +--- + + +# CI/CD pipeline components catalog + +## Summary + +## Goals + +The goal of the CI/CD pipeline components catalog is to make the authoring of pipeline configurations +easier and more efficient. +It does that by providing a way to discover, learn and reuse pipeline constructs. +It also sets a framework for users to collaborate on pipeline constructs so that they can be evolved +and improved over time. + +This blueprint defines the architectural decisions on how to build a CI/CD catalog of pipeline components +leveraging functionalities that exist today at GitLab. +It also sets the long term path for iterations and improvements to the solution. + +## Challenges + +* GitLab CI/CD can have a steep learning curve for new users. Users must read the documentation and + [YAML reference](TODO) to understand how to configure their pipelines. +* Developers are unable to reuse existing CI/CD templates and have been forced to “reinvent the wheel” and write YAML configurations over and over again. +* GitLab [CI templates](TODO) provide users with scaffolding pipeline or jobs for specific purposes. However + versioning them is challenging today due to being shipped with the GitLab instance. +* Users of GitLab CI/CD (pipeline authors) today have their own ad-hoc way to organize shared pipeline + configurations inside their organization. Those tend to be mostly undocumented configurations. +* Only discoverable configurations are GitLab's CI templates. However they don't have any inline documentation + so it becomes harder to know what they do an how to use them without copy-pasting the content in the + editor and read the actual YAML. +* It’s harder to unlock/discover additional GitLab features (CD, security, test, etc.) +* No framework for testing reusable CI configurations. Many configurations are not unit tested against single + changes. + +## Opportunities + +* Having a catalog of pipeline constructs where users can search and find what they need can greatly lower + the bar for new users. +* Customers are already trying to rollout their ad-hoc catalog of shared configurations. We could provide a + standardized way to write, package and share pipeline constructs directly in the product. +* As we implement new pipeline constructs (for example, reusable job steps) they could be items of the + catalog. The catalog can boost the adoption of new constructs. +* The catalog can be a place where we strengthen our relationship with partners, having components offered + and maintained by our partners. +* With discoverability and better versioning mechanism we can have more improvements and better collaboration. +* Competitive landscape is showing the need for such feature + - [R2DevOps](https://r2devops.io) implements a catalog of CI templates for GitLab pipelines. + - [GitHub Actions](https://github.com/features/actions) provides an extensive catalog of reusable job steps. +* TODO: add more + +### Principles + +- Start with the smallest user base. Dogfood the feature for `gitlab-org` and `gitlab-com` groups. + Involve the Engineering Productivity and other groups authoring pipeline configurations to test + and validate our solutions. +- Ensure we can integrate all the feedback gathered, even if that means changing the technical design or + UX. Until we make the feature GA we should have clear expectations with early adopters. +- Reuse existing functionality as much as possible. Don't reinvent the wheel on the initial iterations. + For example: reuse project features like title, description, avatar to build a catalog. +- Allow the catalog to host future types of pipeline constructs and new ways of using them. +- Design the catalog with self-managed support in mind. + +## Proposal + +>> TODO: describe here that we want to focus on the private catalog for now. + +[PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138) + +### Developing components + +To create a new pipeline component a user must: + +- Create a project. Description and avatar are highly recommended to improve discoverability. +- Add a `component.yml` file that contains the CI configuration. This file is mandatory. +- Add a `README.md` that documents the component. What it does, how to use it, how to contribute, etc. + This file is mandatory. +- Add a `.gitlab-ci.yml` to test that the component works as expected. This file is highly recommended. + +### The CI/CD catalog + +Now that the component is created we want to share it with the rest of the organization. +To share a component and make it available in the catalog, it must be published. To publish a component, the +`project owner` toggles a flag in the project's CI/CD settings. +Then, the project owner or any maintainer must create a tag and [release it](TODO). + +Now the catalog shows the component with its first released version. +New versions of the component can be made available in the catalog by creating tags and releases. + +Any published component located inside a top-level namespace is automatically available for all the projects +inside that group (directly or in subgroups). + +The catalog page is available in the menu inside the top-level group so users can see what components +are available for them to use. +A view of the catalog is also available inside the [Pipeline Editor](TODO) to allow users to quickly +search and use components while editing their pipeline configurations. + +### Using components + +#### Status quo + +Today the only pipeline construct that we have is `include:`. This construct fetches a plain text YAML +configuration from a list of supported locations and merges it with the rest of `.gitlab-ci.yml`. +This construct is the most basic form of configuration composition. +However, due to its simplistic implementation, there are some problems with this: + +- `include`s are layers of generic configurations that are applied on top of each other. + The configuration they include could differ from the final result because other included configurations + could technically override some of the keywords. +- `include`s can contain any valid YAML as long as the merged result is a valid CI YAML configuration. + While this is very powerful feature, it can be used to design overly complex CI configurations, making + debugging very hard. For example, one configuration that defines (or overrides) + [top-level variables](TODO) which are then inherited by all the resulting jobs. +- `includes`s are not validated in isolation, only the merged YAML is validated. It's not possible to fully + isolate the evaluation of the included content. + +Despite these downsides, many users are leveraging `include` to break down complex configurations and reuse +some of the snippets in other pipelines. Some organizations at GitLab have entire projects representing +collections of shareable configurations usable via `include`. + +#### First and future iterations + +Our initial plan is for the catalog to support the existing type of free form YAML configurations via +`include` and Ultimate customers to only access their private catalog. + +This context makes the downsides of `include` highlighted above an acceptable risk because it's what +organizations are already doing today. + +In the future, if we allow all users to publish components to a public catalog we would need to first +address these concerns. +We would need to add new pipeline constructs that are safer, more isolated and deterministic. +Then restrict `include`-type of components only to private catalogs. + +```yaml +# -------------- 1st iteration --------------- +include: + - config: my/old-component-form@2.0 # first iteration (follows existing patterns) + +# ------------- future iterations ------------ +require: + - template: gitlab-org/autodevops@1.1 # template (import the component and use what's needed) + as: ado + +test: + stage: test + use: ado.unitest + +dast: + stage: test + workflow: gitlab-org/workflows/dast@1.0 # single job component (fully isolated) + +deploy: + stage: deploy + script: + - echo "starting deployment" + - step: aws/deploy@1.0 # single step component (fully isolated) +``` + +### Iterations + +1. Experimentation phase + - Build an MVC based on the [PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138). Implement it behind a feature flag with `namespace` actor. + - Enable the feature flag only for `gitlab-com` and `gitlab-org` namespaces to initiate the dogfooding. + - Refine the solution and UX based on feedback. + - Find customers to be early adopters of this feature and iterate on their feedback. +1. Design new pipeline constructs (in parallel with other phases) + - Start the technical and design process to work on proposals for new pipeline constructs (steps, workflows, templates). + - Implement new constructs and make the catalog compatible with hosting them. + - Dogfood new constructs and iterate on feedback. + - Release new constructs on private catalogs. +1. Release the private catalog for groups on Ultimate plan. + - Iterate on feedback. +1. Release the public catalog for all GitLab users (prospect feature) + - Publish new versions of GitLab CI templates as components using the new constructs whenever possible. + - Allow self-managed administrators to populate their self-managed catalog by importing/updating + components from GitLab.com or from repository exports. + - Iterate on feedback. + +## Who + +Proposal: + + + +| Role | Who +|------------------------------|-------------------------| +| Author | Fabio Pitino | +| Engineering Leader | ? | +| Product Manager | Dov Hershkovitch | +| Architecture Evolution Coach | Kamil Trzciński | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Leadership | ? | +| Product | Dov Hershkovitch | +| Engineering | ? | + +Domain experts: + +| Area | Who +|------------------------------|------------------------| +| Verify / Pipeline authoring | Avielle Wolfe | +| Verify / Pipeline execution | Fabio Pitino | + + -- GitLab From 0b9fdd5154a3bba6ba52d7ede8b1585eef98d524 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Fri, 5 Aug 2022 16:11:25 +0200 Subject: [PATCH 02/17] Feedback from review and fix static analyzer warnings --- .../ci_pipeline_components/index.md | 57 +++++++++++-------- 1 file changed, 33 insertions(+), 24 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 529a55e6678a47..f3c57b0122ffb5 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -15,45 +15,53 @@ description: 'Create a catalog of shareable pipeline constructs' The goal of the CI/CD pipeline components catalog is to make the authoring of pipeline configurations easier and more efficient. -It does that by providing a way to discover, learn and reuse pipeline constructs. +It does that by providing a way to discover, understand and learn how to reuse pipeline constructs. It also sets a framework for users to collaborate on pipeline constructs so that they can be evolved and improved over time. -This blueprint defines the architectural decisions on how to build a CI/CD catalog of pipeline components +This blueprint defines the architectural guidelines on how to build a CI/CD catalog of pipeline components leveraging functionalities that exist today at GitLab. It also sets the long term path for iterations and improvements to the solution. ## Challenges -* GitLab CI/CD can have a steep learning curve for new users. Users must read the documentation and - [YAML reference](TODO) to understand how to configure their pipelines. -* Developers are unable to reuse existing CI/CD templates and have been forced to “reinvent the wheel” and write YAML configurations over and over again. -* GitLab [CI templates](TODO) provide users with scaffolding pipeline or jobs for specific purposes. However +- GitLab CI/CD can have a steep learning curve for new users. Users must read the documentation and + [YAML reference](../../../ci/yaml/index.md) to understand how to configure their pipelines. +- Developers are struggling to reuse existing CI/CD templates and have been forced to reinvent the wheel and write + YAML configurations over and over again. +- GitLab [CI templates](../../../development/cicd/templates.md#template-directories) provide users with + scaffolding pipeline or jobs for specific purposes. However versioning them is challenging today due to being shipped with the GitLab instance. -* Users of GitLab CI/CD (pipeline authors) today have their own ad-hoc way to organize shared pipeline + See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) for more information. +- Users of GitLab CI/CD (pipeline authors) today have their own ad-hoc way to organize shared pipeline configurations inside their organization. Those tend to be mostly undocumented configurations. -* Only discoverable configurations are GitLab's CI templates. However they don't have any inline documentation +- Only discoverable configurations are GitLab CI templates. However they don't have any inline documentation so it becomes harder to know what they do an how to use them without copy-pasting the content in the editor and read the actual YAML. -* It’s harder to unlock/discover additional GitLab features (CD, security, test, etc.) -* No framework for testing reusable CI configurations. Many configurations are not unit tested against single +- It's harder to unlock/discover additional GitLab features (CD, security, test, etc.) +- No framework for testing reusable CI configurations. Many configurations are not unit tested against single changes. +- Communities, partners, 3rd parties, individual contributors, must go through the + [GitLab Contribution process](https://about.gitlab.com/community/contribute/) to contribute to GitLab managed + templates. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/323727) for more information. +- GitLab has more than 100 of templates with some of them barely maintained after their addition. ## Opportunities -* Having a catalog of pipeline constructs where users can search and find what they need can greatly lower +- Having a catalog of pipeline constructs where users can search and find what they need can greatly lower the bar for new users. -* Customers are already trying to rollout their ad-hoc catalog of shared configurations. We could provide a +- Customers are already trying to rollout their ad-hoc catalog of shared configurations. We could provide a standardized way to write, package and share pipeline constructs directly in the product. -* As we implement new pipeline constructs (for example, reusable job steps) they could be items of the +- As we implement new pipeline constructs (for example, reusable job steps) they could be items of the catalog. The catalog can boost the adoption of new constructs. -* The catalog can be a place where we strengthen our relationship with partners, having components offered +- The catalog can be a place where we strengthen our relationship with partners, having components offered and maintained by our partners. -* With discoverability and better versioning mechanism we can have more improvements and better collaboration. -* Competitive landscape is showing the need for such feature +- With discoverability and better versioning mechanism we can have more improvements and better collaboration. +- Competitive landscape is showing the need for such feature - [R2DevOps](https://r2devops.io) implements a catalog of CI templates for GitLab pipelines. - [GitHub Actions](https://github.com/features/actions) provides an extensive catalog of reusable job steps. -* TODO: add more + +>> TODO: add more ### Principles @@ -88,7 +96,7 @@ To create a new pipeline component a user must: Now that the component is created we want to share it with the rest of the organization. To share a component and make it available in the catalog, it must be published. To publish a component, the `project owner` toggles a flag in the project's CI/CD settings. -Then, the project owner or any maintainer must create a tag and [release it](TODO). +Then, the project owner or any maintainer must create a tag and [release it](../../../user/project/releases/index.md). Now the catalog shows the component with its first released version. New versions of the component can be made available in the catalog by creating tags and releases. @@ -98,8 +106,8 @@ inside that group (directly or in subgroups). The catalog page is available in the menu inside the top-level group so users can see what components are available for them to use. -A view of the catalog is also available inside the [Pipeline Editor](TODO) to allow users to quickly -search and use components while editing their pipeline configurations. +A view of the catalog is also available inside the [Pipeline Editor](../../../ci/pipeline_editor/index.md). +This allow users to quickly search and use components while editing their pipeline configurations. ### Using components @@ -116,7 +124,7 @@ However, due to its simplistic implementation, there are some problems with this - `include`s can contain any valid YAML as long as the merged result is a valid CI YAML configuration. While this is very powerful feature, it can be used to design overly complex CI configurations, making debugging very hard. For example, one configuration that defines (or overrides) - [top-level variables](TODO) which are then inherited by all the resulting jobs. + [top-level variables](../../../ci/yaml/index.md#variables) which are then inherited by all the resulting jobs. - `includes`s are not validated in isolation, only the merged YAML is validated. It's not possible to fully isolate the evaluation of the included content. @@ -132,9 +140,9 @@ Our initial plan is for the catalog to support the existing type of free form YA This context makes the downsides of `include` highlighted above an acceptable risk because it's what organizations are already doing today. -In the future, if we allow all users to publish components to a public catalog we would need to first +In the future, if we allow all users to publish components to a public catalog we must first address these concerns. -We would need to add new pipeline constructs that are safer, more isolated and deterministic. +We must add new pipeline constructs that are safer, more isolated and deterministic. Then restrict `include`-type of components only to private catalogs. ```yaml @@ -165,7 +173,8 @@ deploy: ### Iterations 1. Experimentation phase - - Build an MVC based on the [PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138). Implement it behind a feature flag with `namespace` actor. + - Build an MVC based on the [PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138). + Implement it behind a feature flag with `namespace` actor. - Enable the feature flag only for `gitlab-com` and `gitlab-org` namespaces to initiate the dogfooding. - Refine the solution and UX based on feedback. - Find customers to be early adopters of this feature and iterate on their feedback. -- GitLab From 7197d4a8bf27deda7682a166fcd3e5ee44b3b651 Mon Sep 17 00:00:00 2001 From: Mark Nuzzo Date: Mon, 8 Aug 2022 13:51:54 +0000 Subject: [PATCH 03/17] Apply 6 suggestion(s) to 1 file(s) --- .../blueprints/ci_pipeline_components/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index f3c57b0122ffb5..864153b1771454 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -15,28 +15,28 @@ description: 'Create a catalog of shareable pipeline constructs' The goal of the CI/CD pipeline components catalog is to make the authoring of pipeline configurations easier and more efficient. -It does that by providing a way to discover, understand and learn how to reuse pipeline constructs. -It also sets a framework for users to collaborate on pipeline constructs so that they can be evolved +Providing a way to discover, understand and learn how to reuse pipeline constructs allows for a more streamlined experience. +Having a CI/CD pipeline components catalog also sets a framework for users to collaborate on pipeline constructs so that they can be evolved and improved over time. This blueprint defines the architectural guidelines on how to build a CI/CD catalog of pipeline components leveraging functionalities that exist today at GitLab. -It also sets the long term path for iterations and improvements to the solution. +This blueprint also defines the long-term direction for iterations and improvements to the solution. ## Challenges - GitLab CI/CD can have a steep learning curve for new users. Users must read the documentation and [YAML reference](../../../ci/yaml/index.md) to understand how to configure their pipelines. -- Developers are struggling to reuse existing CI/CD templates and have been forced to reinvent the wheel and write - YAML configurations over and over again. +- Developers are struggling to reuse existing CI/CD templates with the result of having to reinvent the wheel and write + YAML configurations repeatedly. - GitLab [CI templates](../../../development/cicd/templates.md#template-directories) provide users with scaffolding pipeline or jobs for specific purposes. However versioning them is challenging today due to being shipped with the GitLab instance. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) for more information. - Users of GitLab CI/CD (pipeline authors) today have their own ad-hoc way to organize shared pipeline - configurations inside their organization. Those tend to be mostly undocumented configurations. + configurations inside their organization. Those configurations tend to be mostly undocumented. - Only discoverable configurations are GitLab CI templates. However they don't have any inline documentation - so it becomes harder to know what they do an how to use them without copy-pasting the content in the + so it becomes harder to know what they do and how to use them without copy-pasting the content in the editor and read the actual YAML. - It's harder to unlock/discover additional GitLab features (CD, security, test, etc.) - No framework for testing reusable CI configurations. Many configurations are not unit tested against single -- GitLab From a3787b257d3c166824a742800c1e58124151feb9 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 17 Aug 2022 15:49:10 +0200 Subject: [PATCH 04/17] Include the dev workflow and more examples --- .../ci_pipeline_components/index.md | 157 ++++++++++++++++-- 1 file changed, 143 insertions(+), 14 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 864153b1771454..343a316148e9b1 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -86,29 +86,157 @@ This blueprint also defines the long-term direction for iterations and improveme To create a new pipeline component a user must: - Create a project. Description and avatar are highly recommended to improve discoverability. -- Add a `component.yml` file that contains the CI configuration. This file is mandatory. -- Add a `README.md` that documents the component. What it does, how to use it, how to contribute, etc. +- Create a nested directory `.gitlab/ci`. +- Add a `.gitlab/ci/component.yml` file that contains the CI configuration. This file is mandatory. +- Add a `README.md` in the top level directory that documents the component. + What it does, how to use it, how to contribute, etc. This file is mandatory. -- Add a `.gitlab-ci.yml` to test that the component works as expected. This file is highly recommended. +- Add a `.gitlab-ci.yml` in the top level directory to test that the component works as expected. + This file is highly recommended. + +The `.gitlab/ci/component.yml` file contains the configuration to be used for the particular type. +Initially we support the free form of configuration that can be used with `include:`. Later we can +support more types. + +>> Note: the format is not finalized because it requires solution validation and dogfooding. + +Example of free form configuration: + +```yaml +# `config` here is the top-level keyword that indicates the type +# being used. Later we can support types such as `workflow:` and +# `steps:` +# The content of `config` is evaluated as-is. +config: + variables: + FOO: foo + BAR: bar + + build: + script: echo $FOO + stage: build + + test: + script: echo $BAR + stage: test +``` + +Later we can support types such as `workflow` and `steps`. Example of a possible syntax: + +```yaml +workflow: # A workflow runs as a child pipeline + build: + script: echo build + stage: build + test: + script: echo test + stage: test +``` + +```yaml +steps: + build: + script: echo build # stage will be specified where it's used. + test: + script: echo build +``` + +Or a combination of multiple types: + +```yaml +# showing how anchors could be used to reuse portions of configuration +# across types. +.steps_build: &build + script: echo build +.steps_test: &test + script: echo test + +workflow: + build: + stage: build + <<: *build + test: + stage: test + <<: *test + +steps: + build: + <<: *build + test: + <<: *test +``` + +Alternative approaches discussed: + +- use a file per type: `.gitlab/ci/steps/build.yml`, `.gitlab/ci/workflows/.gitlab-ci.yml`. + - Pros: better organization of types + - Cons: Must use `include:` or similar methods to reuse common patterns, incurring in more + Gitaly calls. + +### Testing components + +The `.gitlab-ci.yml` file in the top-level directory of the repository is the standard location +GitLab uses to create a pipeline on changes to the repository. +We can use this file to ensure that the component works after every change. + +```yaml +# project: gitlab-org/components/dast +# file: .gitlab-ci.yml + +include: + # equivalent to: + # config: gitlab-org/components/dast@ + - config: $CI_PROJECT_PATH@$CI_COMMIT_SHA + +other-job: + stage: test + script: echo "do something" +``` + +The syntax loads the configuration file `.gitlab/ci/component.yml` from `gitlab-org/components/dast` project +at the given commit SHA. +Then the configuration is evaluated and expanded before being merged to the main `.gitlab-ci.yml` file. + +### Component path + +The component is identified by the project full path and the version: + +```plaintext +gitlab-org/components/dast@1.0 +gitlab-org/components/dast@d96f40f0dbece8004206cd8b0410407350fb2dd6 +``` + +The supported version can either be: + +- a released tag - the official versioning mechanism. +- a commit SHA - can be used for testing unreleased versions. ### The CI/CD catalog Now that the component is created we want to share it with the rest of the organization. -To share a component and make it available in the catalog, it must be published. To publish a component, the -`project owner` toggles a flag in the project's CI/CD settings. +To share a component and make it available in the catalog, it must be marked as a "CI component". +To do that, the `project owner` toggles a flag in the project's CI/CD settings. Then, the project owner or any maintainer must create a tag and [release it](../../../user/project/releases/index.md). Now the catalog shows the component with its first released version. -New versions of the component can be made available in the catalog by creating tags and releases. +New versions can be added by creating a tag and release it. New releases are automatically available in the catalog. -Any published component located inside a top-level namespace is automatically available for all the projects +Any component located inside a top-level namespace is automatically available for all the projects inside that group (directly or in subgroups). +For example: `gitlab-org/components/dast` can be used by any projects inside `gitlab-org`. + +In future iterations we allowing a component to be made publicly available so that any projects on GitLab can +use it. + The catalog page is available in the menu inside the top-level group so users can see what components are available for them to use. A view of the catalog is also available inside the [Pipeline Editor](../../../ci/pipeline_editor/index.md). This allow users to quickly search and use components while editing their pipeline configurations. +Originally we build catalog views on top of the existing [Projects Explorer](https://gitlab.com/explore) implementation. +Over time we evaluate whether to build an entirely separate view. + ### Using components #### Status quo @@ -134,8 +262,8 @@ collections of shareable configurations usable via `include`. #### First and future iterations -Our initial plan is for the catalog to support the existing type of free form YAML configurations via -`include` and Ultimate customers to only access their private catalog. +The first iteration is for the catalog to support the existing type of free form YAML configurations via +`include`. The private CI components catalog feature is available only to Ultimate customers. This context makes the downsides of `include` highlighted above an acceptable risk because it's what organizations are already doing today. @@ -151,17 +279,18 @@ include: - config: my/old-component-form@2.0 # first iteration (follows existing patterns) # ------------- future iterations ------------ -require: - - template: gitlab-org/autodevops@1.1 # template (import the component and use what's needed) - as: ado +import: + - ado: gitlab-org/autodevops@1.1 # template (import the component and use what's needed) test: stage: test - use: ado.unitest + use: ado/unitest # or leverage `extends` dast: stage: test - workflow: gitlab-org/workflows/dast@1.0 # single job component (fully isolated) + trigger: + workflow: gitlab-org/workflows/dast@1.0 # single job component (fully isolated) + strategy: depend deploy: stage: deploy -- GitLab From 376d2e87a1fe213c714fa8b407fec9a93d415333 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 17 Aug 2022 16:37:18 +0200 Subject: [PATCH 05/17] Add characteristics of a component --- .../ci_pipeline_components/index.md | 53 +++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 343a316148e9b1..2d49d3d508135f 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -63,7 +63,7 @@ This blueprint also defines the long-term direction for iterations and improveme >> TODO: add more -### Principles +## Implementation guidelines - Start with the smallest user base. Dogfood the feature for `gitlab-org` and `gitlab-com` groups. Involve the Engineering Productivity and other groups authoring pipeline configurations to test @@ -72,8 +72,22 @@ This blueprint also defines the long-term direction for iterations and improveme UX. Until we make the feature GA we should have clear expectations with early adopters. - Reuse existing functionality as much as possible. Don't reinvent the wheel on the initial iterations. For example: reuse project features like title, description, avatar to build a catalog. -- Allow the catalog to host future types of pipeline constructs and new ways of using them. +- Leverage GitLab features for the development lifecycle of the components (testing via `.gitlab-ci.yml`, + release management, Pipeline Editor, etc.). - Design the catalog with self-managed support in mind. +- Allow the catalog an the workflow to support future types of pipeline constructs and new ways of using them. + +## Characteristics of a component + +For best experience with any systems made of components it's fundamental that components are single purposed, +isolated and reusable. + +- **Single purpose**: a component must focus on a single goal and the scope be as small as possible. +- **Isolation**: when a component is used in a pipeline, its implementation details should not leak outside the + component itself and into the main pipeline. +- **Reusability:** a component is designed to be used in different pipelines. + Depending on the assumptions it's built on a component can be more or less generic. + Generic components are more reusable but may require more customization. ## Proposal @@ -164,7 +178,7 @@ steps: <<: *build test: <<: *test -``` +``` Alternative approaches discussed: @@ -299,6 +313,39 @@ deploy: - step: aws/deploy@1.0 # single step component (fully isolated) ``` +To follow the [characteristics of a component](#characteristics-of-a-component), the initial component type +used through `include:config` should differ from other types of `include`: + +- Any top-level keywords (default job attributes) should be merged into each component's job prior to using + the component. The top-level keywords are removed before merging so they don't leak into the main pipeline. + Doing this avoids any name conflicts on variables as well as evaluating the component in a sandbox. + +```yaml +# content of the component +variables: + FOO: foo + BAR: bar + +job_a: + stage: test + script: echo $FOO + +job_b: + stage: test + script: echo $BAR + +# --------------------- +# component being used: +include: + - config: org/my-component@1.0 + +test: + script: echo $FOO # variable must be empty +``` + +- Any hidden configurations (`.my-template`) should be removed prior to merge the content with the main pipeline. + Doing this avoids any name conflicts on hidden configurations. + ### Iterations 1. Experimentation phase -- GitLab From a33134c8fbb5f5d6ebb09812989e2fb759d0895b Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Tue, 23 Aug 2022 17:14:40 +0200 Subject: [PATCH 06/17] Add Nadia as UX DRI --- .../blueprints/ci_pipeline_components/index.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 2d49d3d508135f..ea353f380a5d63 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -243,9 +243,12 @@ For example: `gitlab-org/components/dast` can be used by any projects inside `gi In future iterations we allowing a component to be made publicly available so that any projects on GitLab can use it. -The catalog page is available in the menu inside the top-level group so users can see what components -are available for them to use. -A view of the catalog is also available inside the [Pipeline Editor](../../../ci/pipeline_editor/index.md). +The catalog page is available from + +- The top-level group so users can see what components are available for them to use: **group > CI/CD > CI/CD resources**. +- The project so users can see what components the project can use: **project > CI/CD > CI/CD resources**. + +In future iteration we add a view of the catalog inside the [Pipeline Editor](../../../ci/pipeline_editor/index.md). This allow users to quickly search and use components while editing their pipeline configurations. Originally we build catalog views on top of the existing [Projects Explorer](https://gitlab.com/explore) implementation. @@ -387,6 +390,7 @@ DRIs: | Leadership | ? | | Product | Dov Hershkovitch | | Engineering | ? | +| UX | Nadia Sotnikova | Domain experts: -- GitLab From 4afecca9f36f391f220b20c12330c792ceb97b92 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Thu, 25 Aug 2022 13:29:37 +0200 Subject: [PATCH 07/17] Describe problems with GitLab CI templates --- .../ci_pipeline_components/index.md | 32 +++++++++++++++---- 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index ea353f380a5d63..79b7568fafaab7 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -30,22 +30,42 @@ This blueprint also defines the long-term direction for iterations and improveme - Developers are struggling to reuse existing CI/CD templates with the result of having to reinvent the wheel and write YAML configurations repeatedly. - GitLab [CI templates](../../../development/cicd/templates.md#template-directories) provide users with - scaffolding pipeline or jobs for specific purposes. However - versioning them is challenging today due to being shipped with the GitLab instance. + scaffolding pipeline or jobs for specific purposes. + However versioning them is challenging today due to being shipped with the GitLab instance. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/17716) for more information. - Users of GitLab CI/CD (pipeline authors) today have their own ad-hoc way to organize shared pipeline configurations inside their organization. Those configurations tend to be mostly undocumented. -- Only discoverable configurations are GitLab CI templates. However they don't have any inline documentation +- The only discoverable configurations are GitLab CI templates. However they don't have any inline documentation so it becomes harder to know what they do and how to use them without copy-pasting the content in the editor and read the actual YAML. -- It's harder to unlock/discover additional GitLab features (CD, security, test, etc.) -- No framework for testing reusable CI configurations. Many configurations are not unit tested against single - changes. +- It's harder to adopt additional GitLab features (CD, security, test, etc.). +- There is no framework for testing reusable CI configurations. + Many configurations are not unit tested against single changes. - Communities, partners, 3rd parties, individual contributors, must go through the [GitLab Contribution process](https://about.gitlab.com/community/contribute/) to contribute to GitLab managed templates. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/323727) for more information. - GitLab has more than 100 of templates with some of them barely maintained after their addition. +### Problems with GitLab CI templates + +- `Jobs/` templates hard-code the `stage:` attribute but the user of the template must somehow override + or know in advance what stage is needed. + - The user should be able to import the job inside a given stage or pass the stage names as input parameter + when using the component. + - Failures in mapping the correct stage can result in confusing errors. +- Some templates are designed to work with AutoDevops but are not generic enough + ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/2c0e8e4470001442e999391df81e19732b3439e6/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml)). +- Many CI templates, especially those [language specific](https://gitlab.com/gitlab-org/gitlab/-/tree/2c0e8e4470001442e999391df81e19732b3439e6/lib/gitlab/ci/templates) + are tutorial/scaffolding-style templates. + - They are meant to show the user how a typical pipeline would look like but it requires high customization from the user perspective. + - They require a different UX: copy-paste in the position of the Pipeline Editor cursor. +- Some templates like `SAST.latest.gitlab-ci.yml` add multiple jobs conditionally to the same pipeline. + - Ideally these jobs could run as a child pipeline and make the reports available to the parent pipeline. + - [This epic](https://gitlab.com/groups/gitlab-org/-/epics/8205) is necessary for Parent-child pipelines to be used. +- Some templates incorrectly use `variables`, `image` and other top-level keywords but that defines them in all pipeline jobs, + not just those defined in the template. + - This technique introduces inheritance issues when a template modifies jobs unnecessarily. + ## Opportunities - Having a catalog of pipeline constructs where users can search and find what they need can greatly lower -- GitLab From 1cc14e4efb30bf1dc5c878f7affd3a56459f2fc5 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Thu, 25 Aug 2022 13:49:31 +0200 Subject: [PATCH 08/17] Add Glossary section --- .../blueprints/ci_pipeline_components/index.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 79b7568fafaab7..4292319928478e 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -97,6 +97,18 @@ This blueprint also defines the long-term direction for iterations and improveme - Design the catalog with self-managed support in mind. - Allow the catalog an the workflow to support future types of pipeline constructs and new ways of using them. +## Glossary + +This section defines some terms that are used throughout this document. With these terms we are only +identifying abstract concepts and are subject to changes as we refine the design by discovering new insights. + +- **Component** Is the reusable unit of pipeline configuration. +- **Project** Is the GitLab project attached to a repository. A project can contain multiple components. +- **Catalog** is the collection of projects that are set to contain components. +- **Version** is the release name of a tag in the project, which allows components to be pinned to a specific revision. + +>> TODO: find a term for a "project hosting components". Before we called it "component" when there was a 1-1 relation between component and project. + ## Characteristics of a component For best experience with any systems made of components it's fundamental that components are single purposed, -- GitLab From 4d2d797010c83b6c31032b7be0e72e1b85f3600a Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Thu, 25 Aug 2022 14:45:37 +0200 Subject: [PATCH 09/17] Consolidate new syntax --- .../ci_pipeline_components/index.md | 216 ++++++++++-------- 1 file changed, 119 insertions(+), 97 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 4292319928478e..4c8e615b2805a2 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -108,6 +108,7 @@ identifying abstract concepts and are subject to changes as we refine the design - **Version** is the release name of a tag in the project, which allows components to be pinned to a specific revision. >> TODO: find a term for a "project hosting components". Before we called it "component" when there was a 1-1 relation between component and project. +>> We should probably refer as "component" the smallest unit available from `spec:export:`. ## Characteristics of a component @@ -140,109 +141,117 @@ To create a new pipeline component a user must: - Add a `.gitlab-ci.yml` in the top level directory to test that the component works as expected. This file is highly recommended. -The `.gitlab/ci/component.yml` file contains the configuration to be used for the particular type. -Initially we support the free form of configuration that can be used with `include:`. Later we can -support more types. +The `.gitlab/ci/component.yml` file defines the specification of a component, what configurations +it exposes and their types, what inputs are defined for each configuration, etc. ->> Note: the format is not finalized because it requires solution validation and dogfooding. +>> Note: The format is not finalized because it requires a POC as well as solution validation and dogfooding. -Example of free form configuration: +Example of `component.yml` file for DAST template: ```yaml -# `config` here is the top-level keyword that indicates the type -# being used. Later we can support types such as `workflow:` and -# `steps:` -# The content of `config` is evaluated as-is. -config: - variables: - FOO: foo - BAR: bar - - build: - script: echo $FOO - stage: build - - test: - script: echo $BAR - stage: test +# Top level keyword indicating the specifications of this component. +spec: + # This section defines what other components this one depends on. + import: + another-component: org/another-component@1.0 + + # This section defines what configurations this component provides. + # A single file can describe multiple configurations. + export: + # name of the configuration exported by the component. + # this is like a public function exported by a libarary. + all-jobs: + kind: config # this specifies how this function can be used. + inputs: # parameters that this function depends on. + website: + required: true + stage: + default: test + # this content is evaluated based on the `kind:`. As in this case it's a + # `config` type, it can contain any CI YAML configuration that can be + # used with `include:`. + content: + scan-x: + stage: $stage # use of inputs + script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x + scan-y: + stage: $stage + script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=y + + scan-x: + kind: job + inputs: + website: + required: true + content: + script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x ``` -Later we can support types such as `workflow` and `steps`. Example of a possible syntax: +As MVC we can support only `config` type to be used with `include:` but later we can support +other types such as `job` template, `workflow` (run a configuration as a child pipeline), job `steps`. -```yaml -workflow: # A workflow runs as a child pipeline - build: - script: echo build - stage: build - test: - script: echo test - stage: test -``` - -```yaml -steps: - build: - script: echo build # stage will be specified where it's used. - test: - script: echo build -``` - -Or a combination of multiple types: +Being a standard YAML file we could also leverage the YAML anchors to deduplicate part of the +configuration for better maintenance: ```yaml -# showing how anchors could be used to reuse portions of configuration -# across types. -.steps_build: &build - script: echo build -.steps_test: &test - script: echo test - -workflow: - build: - stage: build - <<: *build - test: - stage: test - <<: *test - -steps: - build: - <<: *build - test: - <<: *test +.base-inputs: &base-inputs + website: + required: true + +.scan-x-script: &scan-x-script + script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x + +spec: + export: + all-jobs: + kind: config + inputs: + <<: *base-inputs + stage: + default: test + content: + scan-x: + stage: $stage + <<: *scan-x-script + scan-y: + stage: $stage + script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=y + + scan-x: + kind: job + inputs: + <<: *base-inputs + content: + <<: *scan-x-script ``` -Alternative approaches discussed: - -- use a file per type: `.gitlab/ci/steps/build.yml`, `.gitlab/ci/workflows/.gitlab-ci.yml`. - - Pros: better organization of types - - Cons: Must use `include:` or similar methods to reuse common patterns, incurring in more - Gitaly calls. - ### Testing components The `.gitlab-ci.yml` file in the top-level directory of the repository is the standard location GitLab uses to create a pipeline on changes to the repository. We can use this file to ensure that the component works after every change. +An example of `.gitlab-ci.yml` for an hypothetically `gitlab-org/dast` project: + ```yaml -# project: gitlab-org/components/dast -# file: .gitlab-ci.yml +# In order to use a component it must be imported first. +import: + # resolves to gitlab-org/dast@ + # The `dast` alias is used later to refer to this component. + dast: $CI_PROJECT_PATH@$CI_COMMIT_SHA +# A specific function of the component can be used based on type. include: - # equivalent to: - # config: gitlab-org/components/dast@ - - config: $CI_PROJECT_PATH@$CI_COMMIT_SHA - -other-job: - stage: test - script: echo "do something" + - config: dast/all-jobs # the `all-jobs` exported configuration for the `dast` component. ``` -The syntax loads the configuration file `.gitlab/ci/component.yml` from `gitlab-org/components/dast` project +The syntax loads the configuration file `.gitlab/ci/component.yml` from `gitlab-org/dast` project at the given commit SHA. Then the configuration is evaluated and expanded before being merged to the main `.gitlab-ci.yml` file. +The `dast/all-jobs` configuration can be used with `include:config` because it's of `kind: config`. +A given configuration kind can only be used in one way. + ### Component path The component is identified by the project full path and the version: @@ -270,12 +279,11 @@ New versions can be added by creating a tag and release it. New releases are aut Any component located inside a top-level namespace is automatically available for all the projects inside that group (directly or in subgroups). -For example: `gitlab-org/components/dast` can be used by any projects inside `gitlab-org`. +For example: `gitlab-org/dast` can be used by any projects inside `gitlab-org`. -In future iterations we allowing a component to be made publicly available so that any projects on GitLab can -use it. +In future iterations we allow a component to be made publicly available so that any projects on GitLab can use it. -The catalog page is available from +The catalog page is available from: - The top-level group so users can see what components are available for them to use: **group > CI/CD > CI/CD resources**. - The project so users can see what components the project can use: **project > CI/CD > CI/CD resources**. @@ -322,30 +330,47 @@ address these concerns. We must add new pipeline constructs that are safer, more isolated and deterministic. Then restrict `include`-type of components only to private catalogs. +For the first iteration we support `kind: config` for the exported component functions which can then +be used with a new `include:config` keyword that can specifically be used with `kind: config` component +functions. + ```yaml -# -------------- 1st iteration --------------- +import: + dast: gitlab-org/dast@1.0 + include: - - config: my/old-component-form@2.0 # first iteration (follows existing patterns) + - config: dast/all-jobs # first iteration (follows existing patterns) +``` -# ------------- future iterations ------------ +In future iterations we can add other kinds of functions: + +- `kind: job` to be use as a job template. +- `kind: workflow` to run a series of jobs as a child pipeline. +- `kind: step` to run a sequence of commands as a job step. + +```yaml import: - - ado: gitlab-org/autodevops@1.1 # template (import the component and use what's needed) + ado: gitlab-org/autodevops@1.0 + dast: gitlab-org/dast@1.0 + deployer: gitlab-org/deployer@1.0 test: stage: test - use: ado/unitest # or leverage `extends` + use: ado/unitest # how we could use functions of `kind: job` -dast: +dast-scans: stage: test trigger: - workflow: gitlab-org/workflows/dast@1.0 # single job component (fully isolated) - strategy: depend + workflow: dast/all-jobs # how we could use functions of `kind: workflow` -deploy: +production: stage: deploy script: - echo "starting deployment" - - step: aws/deploy@1.0 # single step component (fully isolated) + - step: deployer/aws # how we could use functions of `kind: step` + with: + token: $AWS_TOKEN + environment: production ``` To follow the [characteristics of a component](#characteristics-of-a-component), the initial component type @@ -370,15 +395,12 @@ job_b: script: echo $BAR # --------------------- -# component being used: -include: - - config: org/my-component@1.0 - +# where the component is used: test: script: echo $FOO # variable must be empty ``` -- Any hidden configurations (`.my-template`) should be removed prior to merge the content with the main pipeline. +- Any hidden configurations should be removed prior to merge the content with the main pipeline. Doing this avoids any name conflicts on hidden configurations. ### Iterations -- GitLab From 4a34142cf53eb3ea30534ccc661d9546d4f81371 Mon Sep 17 00:00:00 2001 From: Marshall Cottrell Date: Mon, 5 Sep 2022 15:53:38 +0000 Subject: [PATCH 10/17] Apply 1 suggestion(s) to 1 file(s) --- doc/architecture/blueprints/ci_pipeline_components/index.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 4c8e615b2805a2..756339c5e9855c 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -65,6 +65,8 @@ This blueprint also defines the long-term direction for iterations and improveme - Some templates incorrectly use `variables`, `image` and other top-level keywords but that defines them in all pipeline jobs, not just those defined in the template. - This technique introduces inheritance issues when a template modifies jobs unnecessarily. +- Passing artifacts between jobs comes with a lot of overhead. Passing structured data between jobs is essentially impossible. + - By default, artifacts are implicitly passed between jobs impacting performance and scalability ## Opportunities -- GitLab From 8c682cf06a4b81d1c156563eda2fde01ac2e78d7 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 7 Sep 2022 16:43:46 +0100 Subject: [PATCH 11/17] Move proposal to a new MR --- .../ci_pipeline_components/index.md | 297 +----------------- 1 file changed, 14 insertions(+), 283 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 756339c5e9855c..73228233086560 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -83,8 +83,6 @@ This blueprint also defines the long-term direction for iterations and improveme - [R2DevOps](https://r2devops.io) implements a catalog of CI templates for GitLab pipelines. - [GitHub Actions](https://github.com/features/actions) provides an extensive catalog of reusable job steps. ->> TODO: add more - ## Implementation guidelines - Start with the smallest user base. Dogfood the feature for `gitlab-org` and `gitlab-com` groups. @@ -109,9 +107,6 @@ identifying abstract concepts and are subject to changes as we refine the design - **Catalog** is the collection of projects that are set to contain components. - **Version** is the release name of a tag in the project, which allows components to be pinned to a specific revision. ->> TODO: find a term for a "project hosting components". Before we called it "component" when there was a 1-1 relation between component and project. ->> We should probably refer as "component" the smallest unit available from `spec:export:`. - ## Characteristics of a component For best experience with any systems made of components it's fundamental that components are single purposed, @@ -126,296 +121,32 @@ isolated and reusable. ## Proposal ->> TODO: describe here that we want to focus on the private catalog for now. - -[PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138) - -### Developing components - -To create a new pipeline component a user must: - -- Create a project. Description and avatar are highly recommended to improve discoverability. -- Create a nested directory `.gitlab/ci`. -- Add a `.gitlab/ci/component.yml` file that contains the CI configuration. This file is mandatory. -- Add a `README.md` in the top level directory that documents the component. - What it does, how to use it, how to contribute, etc. - This file is mandatory. -- Add a `.gitlab-ci.yml` in the top level directory to test that the component works as expected. - This file is highly recommended. - -The `.gitlab/ci/component.yml` file defines the specification of a component, what configurations -it exposes and their types, what inputs are defined for each configuration, etc. - ->> Note: The format is not finalized because it requires a POC as well as solution validation and dogfooding. - -Example of `component.yml` file for DAST template: - -```yaml -# Top level keyword indicating the specifications of this component. -spec: - # This section defines what other components this one depends on. - import: - another-component: org/another-component@1.0 - - # This section defines what configurations this component provides. - # A single file can describe multiple configurations. - export: - # name of the configuration exported by the component. - # this is like a public function exported by a libarary. - all-jobs: - kind: config # this specifies how this function can be used. - inputs: # parameters that this function depends on. - website: - required: true - stage: - default: test - # this content is evaluated based on the `kind:`. As in this case it's a - # `config` type, it can contain any CI YAML configuration that can be - # used with `include:`. - content: - scan-x: - stage: $stage # use of inputs - script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x - scan-y: - stage: $stage - script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=y - - scan-x: - kind: job - inputs: - website: - required: true - content: - script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x -``` - -As MVC we can support only `config` type to be used with `include:` but later we can support -other types such as `job` template, `workflow` (run a configuration as a child pipeline), job `steps`. - -Being a standard YAML file we could also leverage the YAML anchors to deduplicate part of the -configuration for better maintenance: - -```yaml -.base-inputs: &base-inputs - website: - required: true - -.scan-x-script: &scan-x-script - script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=x - -spec: - export: - all-jobs: - kind: config - inputs: - <<: *base-inputs - stage: - default: test - content: - scan-x: - stage: $stage - <<: *scan-x-script - scan-y: - stage: $stage - script: ./dast --website=$website --project=$CI_PROJECT_PATH --profile=y - - scan-x: - kind: job - inputs: - <<: *base-inputs - content: - <<: *scan-x-script -``` - -### Testing components - -The `.gitlab-ci.yml` file in the top-level directory of the repository is the standard location -GitLab uses to create a pipeline on changes to the repository. -We can use this file to ensure that the component works after every change. - -An example of `.gitlab-ci.yml` for an hypothetically `gitlab-org/dast` project: - -```yaml -# In order to use a component it must be imported first. -import: - # resolves to gitlab-org/dast@ - # The `dast` alias is used later to refer to this component. - dast: $CI_PROJECT_PATH@$CI_COMMIT_SHA - -# A specific function of the component can be used based on type. -include: - - config: dast/all-jobs # the `all-jobs` exported configuration for the `dast` component. -``` - -The syntax loads the configuration file `.gitlab/ci/component.yml` from `gitlab-org/dast` project -at the given commit SHA. -Then the configuration is evaluated and expanded before being merged to the main `.gitlab-ci.yml` file. - -The `dast/all-jobs` configuration can be used with `include:config` because it's of `kind: config`. -A given configuration kind can only be used in one way. - -### Component path - -The component is identified by the project full path and the version: - -```plaintext -gitlab-org/components/dast@1.0 -gitlab-org/components/dast@d96f40f0dbece8004206cd8b0410407350fb2dd6 -``` - -The supported version can either be: - -- a released tag - the official versioning mechanism. -- a commit SHA - can be used for testing unreleased versions. - -### The CI/CD catalog - -Now that the component is created we want to share it with the rest of the organization. -To share a component and make it available in the catalog, it must be marked as a "CI component". -To do that, the `project owner` toggles a flag in the project's CI/CD settings. -Then, the project owner or any maintainer must create a tag and [release it](../../../user/project/releases/index.md). - -Now the catalog shows the component with its first released version. -New versions can be added by creating a tag and release it. New releases are automatically available in the catalog. - -Any component located inside a top-level namespace is automatically available for all the projects -inside that group (directly or in subgroups). - -For example: `gitlab-org/dast` can be used by any projects inside `gitlab-org`. - -In future iterations we allow a component to be made publicly available so that any projects on GitLab can use it. - -The catalog page is available from: - -- The top-level group so users can see what components are available for them to use: **group > CI/CD > CI/CD resources**. -- The project so users can see what components the project can use: **project > CI/CD > CI/CD resources**. - -In future iteration we add a view of the catalog inside the [Pipeline Editor](../../../ci/pipeline_editor/index.md). -This allow users to quickly search and use components while editing their pipeline configurations. - -Originally we build catalog views on top of the existing [Projects Explorer](https://gitlab.com/explore) implementation. -Over time we evaluate whether to build an entirely separate view. - -### Using components - -#### Status quo - -Today the only pipeline construct that we have is `include:`. This construct fetches a plain text YAML -configuration from a list of supported locations and merges it with the rest of `.gitlab-ci.yml`. -This construct is the most basic form of configuration composition. -However, due to its simplistic implementation, there are some problems with this: - -- `include`s are layers of generic configurations that are applied on top of each other. - The configuration they include could differ from the final result because other included configurations - could technically override some of the keywords. -- `include`s can contain any valid YAML as long as the merged result is a valid CI YAML configuration. - While this is very powerful feature, it can be used to design overly complex CI configurations, making - debugging very hard. For example, one configuration that defines (or overrides) - [top-level variables](../../../ci/yaml/index.md#variables) which are then inherited by all the resulting jobs. -- `includes`s are not validated in isolation, only the merged YAML is validated. It's not possible to fully - isolate the evaluation of the included content. - -Despite these downsides, many users are leveraging `include` to break down complex configurations and reuse -some of the snippets in other pipelines. Some organizations at GitLab have entire projects representing -collections of shareable configurations usable via `include`. - -#### First and future iterations - -The first iteration is for the catalog to support the existing type of free form YAML configurations via -`include`. The private CI components catalog feature is available only to Ultimate customers. - -This context makes the downsides of `include` highlighted above an acceptable risk because it's what -organizations are already doing today. - -In the future, if we allow all users to publish components to a public catalog we must first -address these concerns. -We must add new pipeline constructs that are safer, more isolated and deterministic. -Then restrict `include`-type of components only to private catalogs. - -For the first iteration we support `kind: config` for the exported component functions which can then -be used with a new `include:config` keyword that can specifically be used with `kind: config` component -functions. - -```yaml -import: - dast: gitlab-org/dast@1.0 - -include: - - config: dast/all-jobs # first iteration (follows existing patterns) -``` - -In future iterations we can add other kinds of functions: - -- `kind: job` to be use as a job template. -- `kind: workflow` to run a series of jobs as a child pipeline. -- `kind: step` to run a sequence of commands as a job step. - -```yaml -import: - ado: gitlab-org/autodevops@1.0 - dast: gitlab-org/dast@1.0 - deployer: gitlab-org/deployer@1.0 - -test: - stage: test - use: ado/unitest # how we could use functions of `kind: job` +>> placeholder -dast-scans: - stage: test - trigger: - workflow: dast/all-jobs # how we could use functions of `kind: workflow` +## Limits -production: - stage: deploy - script: - - echo "starting deployment" - - step: deployer/aws # how we could use functions of `kind: step` - with: - token: $AWS_TOKEN - environment: production -``` +Any MVC that exposes a feature should be added with limitations from the beginning. +It's safer to add new features with restrictions than trying to limit a feature after it's being used. +We can always soften the restrictions later depending on user demand. -To follow the [characteristics of a component](#characteristics-of-a-component), the initial component type -used through `include:config` should differ from other types of `include`: +Some limits we could consider adding: -- Any top-level keywords (default job attributes) should be merged into each component's job prior to using - the component. The top-level keywords are removed before merging so they don't leak into the main pipeline. - Doing this avoids any name conflicts on variables as well as evaluating the component in a sandbox. +- number of components that a single project can contain/export +- number of imports that a `.gitlab-ci.yml` file can use +- number of imports that a component can declare/use +- max level of nested imports +- max length of the exported component name -```yaml -# content of the component -variables: - FOO: foo - BAR: bar - -job_a: - stage: test - script: echo $FOO - -job_b: - stage: test - script: echo $BAR - -# --------------------- -# where the component is used: -test: - script: echo $FOO # variable must be empty -``` - -- Any hidden configurations should be removed prior to merge the content with the main pipeline. - Doing this avoids any name conflicts on hidden configurations. - -### Iterations +## Iterations 1. Experimentation phase - - Build an MVC based on the [PoC merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81138). - Implement it behind a feature flag with `namespace` actor. + - Build an MVC behind a feature flag with `namespace` actor. - Enable the feature flag only for `gitlab-com` and `gitlab-org` namespaces to initiate the dogfooding. - Refine the solution and UX based on feedback. - Find customers to be early adopters of this feature and iterate on their feedback. 1. Design new pipeline constructs (in parallel with other phases) - Start the technical and design process to work on proposals for new pipeline constructs (steps, workflows, templates). - - Implement new constructs and make the catalog compatible with hosting them. + - Implement new constructs. The catalog must be compatible with them. - Dogfood new constructs and iterate on feedback. - Release new constructs on private catalogs. 1. Release the private catalog for groups on Ultimate plan. -- GitLab From d306bfc1adc6cb826ec900b8e086045247264ef0 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 7 Sep 2022 17:14:37 +0100 Subject: [PATCH 12/17] Add minimal proposal --- .../ci_pipeline_components/index.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 73228233086560..95eff177ef46d1 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -121,7 +121,25 @@ isolated and reusable. ## Proposal ->> placeholder +Prerequisites to create a component: + +- Create a project. Description and avatar are highly recommended to improve discoverability. +- Create a nested directory `.gitlab/ci`. Component files are defined in this directory. +- Add a `README.md` in the top level directory that documents the component. + What it does, how to use it, how to contribute, etc. + This file is mandatory. +- Add a `.gitlab-ci.yml` in the top level directory to test that the component works as expected. + This file is highly recommended. + +Characteristics of a component: + +- It must have a **name** to be referenced to and **description** for extra details. +- It must specify its **type** which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). +- It must define its **content** based on the type. +- It must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. +- Its YAML specification should be **validated statically** (for example: using JSON schema validators). +- It should be possible to use specific **versions** of a component by referencing official releases and SHA. +- It should be possible to use components defined locally in the same repository. ## Limits -- GitLab From 30f1d82e9f79506efb902e39d19f9f9fdca66d28 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Thu, 8 Sep 2022 13:47:48 +0000 Subject: [PATCH 13/17] Add Furkan as Domain Expert. --- doc/architecture/blueprints/ci_pipeline_components/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 95eff177ef46d1..78596cfe78faf1 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -202,6 +202,7 @@ Domain experts: | Area | Who |------------------------------|------------------------| | Verify / Pipeline authoring | Avielle Wolfe | +| Verify / Pipeline authoring | Furkan Ayhan | | Verify / Pipeline execution | Fabio Pitino | -- GitLab From 5e63b1c4efed4ee264b884527bfaf873d5f960a8 Mon Sep 17 00:00:00 2001 From: Grzegorz Bizon Date: Fri, 9 Sep 2022 13:46:22 +0000 Subject: [PATCH 14/17] Apply 6 suggestion(s) to 1 file(s) --- .../blueprints/ci_pipeline_components/index.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 78596cfe78faf1..fed9941bfc7d8b 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -13,14 +13,13 @@ description: 'Create a catalog of shareable pipeline constructs' ## Goals -The goal of the CI/CD pipeline components catalog is to make the authoring of pipeline configurations +The goal of the CI/CD pipeline components catalog is to make the reusing pipeline configurations easier and more efficient. Providing a way to discover, understand and learn how to reuse pipeline constructs allows for a more streamlined experience. Having a CI/CD pipeline components catalog also sets a framework for users to collaborate on pipeline constructs so that they can be evolved and improved over time. -This blueprint defines the architectural guidelines on how to build a CI/CD catalog of pipeline components -leveraging functionalities that exist today at GitLab. +This blueprint defines the architectural guidelines on how to build a CI/CD catalog of pipeline components. This blueprint also defines the long-term direction for iterations and improvements to the solution. ## Challenges @@ -48,6 +47,8 @@ This blueprint also defines the long-term direction for iterations and improveme ### Problems with GitLab CI templates +- GitLab CI Templates have not been designed with deterministic behavior in mind. +- GitLab CI Templates have not been design with reusability in mind. - `Jobs/` templates hard-code the `stage:` attribute but the user of the template must somehow override or know in advance what stage is needed. - The user should be able to import the job inside a given stage or pass the stage names as input parameter @@ -109,8 +110,8 @@ identifying abstract concepts and are subject to changes as we refine the design ## Characteristics of a component -For best experience with any systems made of components it's fundamental that components are single purposed, -isolated and reusable. +For best experience with any systems made of components it's fundamental that components are single purpose, +isolated, reusable and resolvable. - **Single purpose**: a component must focus on a single goal and the scope be as small as possible. - **Isolation**: when a component is used in a pipeline, its implementation details should not leak outside the @@ -118,6 +119,7 @@ isolated and reusable. - **Reusability:** a component is designed to be used in different pipelines. Depending on the assumptions it's built on a component can be more or less generic. Generic components are more reusable but may require more customization. +- **Resolvable:** When a component depends on another component, this dependency needs to be explicit and trackable. Hidden dependencies can lead to myriads of problems. ## Proposal @@ -128,7 +130,7 @@ Prerequisites to create a component: - Add a `README.md` in the top level directory that documents the component. What it does, how to use it, how to contribute, etc. This file is mandatory. -- Add a `.gitlab-ci.yml` in the top level directory to test that the component works as expected. +- Add a `.gitlab-ci.yml` in the top level directory to test that the components works as expected. This file is highly recommended. Characteristics of a component: -- GitLab From 319b299ca4893f67b66da98e79b7dadc2c57b5f3 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Fri, 9 Sep 2022 13:47:44 +0000 Subject: [PATCH 15/17] Apply 1 suggestion(s) to 1 file(s) --- doc/architecture/blueprints/ci_pipeline_components/index.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index fed9941bfc7d8b..7bb11efa062d96 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -66,8 +66,6 @@ This blueprint also defines the long-term direction for iterations and improveme - Some templates incorrectly use `variables`, `image` and other top-level keywords but that defines them in all pipeline jobs, not just those defined in the template. - This technique introduces inheritance issues when a template modifies jobs unnecessarily. -- Passing artifacts between jobs comes with a lot of overhead. Passing structured data between jobs is essentially impossible. - - By default, artifacts are implicitly passed between jobs impacting performance and scalability ## Opportunities -- GitLab From f7f49dc7ea9da975ed2d60563c54aaa85f538bf1 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Fri, 9 Sep 2022 14:42:12 +0000 Subject: [PATCH 16/17] Apply 3 suggestion(s) to 1 file(s) --- doc/architecture/blueprints/ci_pipeline_components/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 7bb11efa062d96..44752a16aafec5 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -95,7 +95,7 @@ This blueprint also defines the long-term direction for iterations and improveme release management, Pipeline Editor, etc.). - Design the catalog with self-managed support in mind. - Allow the catalog an the workflow to support future types of pipeline constructs and new ways of using them. - +- Design components and catalog following industry best practice related to building deterministic package managers. ## Glossary This section defines some terms that are used throughout this document. With these terms we are only @@ -124,7 +124,6 @@ isolated, reusable and resolvable. Prerequisites to create a component: - Create a project. Description and avatar are highly recommended to improve discoverability. -- Create a nested directory `.gitlab/ci`. Component files are defined in this directory. - Add a `README.md` in the top level directory that documents the component. What it does, how to use it, how to contribute, etc. This file is mandatory. @@ -137,6 +136,7 @@ Characteristics of a component: - It must specify its **type** which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). - It must define its **content** based on the type. - It must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. +- It can optionally define **output data** that it returns. - Its YAML specification should be **validated statically** (for example: using JSON schema validators). - It should be possible to use specific **versions** of a component by referencing official releases and SHA. - It should be possible to use components defined locally in the same repository. -- GitLab From 52566853a2d7be64b2e0891e1ba236be64d4488d Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Tue, 13 Sep 2022 15:27:42 +0100 Subject: [PATCH 17/17] Fix static analyzer offense --- doc/architecture/blueprints/ci_pipeline_components/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 44752a16aafec5..94ec3e2f894fc5 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -96,6 +96,7 @@ This blueprint also defines the long-term direction for iterations and improveme - Design the catalog with self-managed support in mind. - Allow the catalog an the workflow to support future types of pipeline constructs and new ways of using them. - Design components and catalog following industry best practice related to building deterministic package managers. + ## Glossary This section defines some terms that are used throughout this document. With these terms we are only -- GitLab