From edb59a0aa76348744e6cf2f12f7872e267472d80 Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Thu, 6 Apr 2023 13:41:45 +0200 Subject: [PATCH 01/13] Starts docs for CI Catalog --- doc/ci/components/index.md | 277 +++++++++++++++++++++++++++++++++++++ 1 file changed, 277 insertions(+) create mode 100644 doc/ci/components/index.md diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md new file mode 100644 index 00000000000000..0c071df4073e4e --- /dev/null +++ b/doc/ci/components/index.md @@ -0,0 +1,277 @@ +--- +stage: Verify +group: Pipeline Composition +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +type: reference +--- + +# CI/CD Catalog (Alpha) + +> - [Introduced]() in Gitlab 15.11. + +This feature is in Alpha and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. +Tell us about your use case by leaving comments in the epic. + +NOTE: This feature is in Alpha. In future releases, to accomodate new features, it might change without notice. + +---- + +The CI/CD Catalog is an index of resources that you can leverage in CI/CD. It contains a list of components repositories that can be discovered and used in pipelines. + +See an example of [development workflow]() for a component repository. + +## Components Repository + +A components repository is a GitLab project/repository that exclusively hosts one or more pipeline components. + +**Structure of a components repository** +A components repository can host one or more components. + +The following directory structure would support 1 component in a repository with the name `rails-rspec`: + +```yaml +├── template.yml +├── README.md +└── .gitlab-ci.yml +``` + +- The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. +- The component is identified by the path `gitlab.com/gitlab-org/rails-rspec`, which is the project path. +- We expect a `template.yml` file and `README.md` to be located in the root directory of the repository. + +The following directory structure would support multiple components per repository: + + +```yaml +├── .gitlab-ci.yml +├── README.md +├── unit/ +│ └── template.yml +├── integration/ +│ └── template.yml +└── feature/ + └── template.yml +``` + +Each of these components are identified by their path `gitlab.com/gitlab-org/rails-rspec/unit`, `gitlab.com/myorg/rails-rspec/integration` +and `gitlab.com/myorg/rails-rspec/feature`. + +This directory structure could also support both strategies: + +```yaml +. +├── template.yml +├── README.md +├── LICENSE +├── .gitlab-ci.yml +├── unit/ +│ └── template.yml +├── integration/ +│ └── template.yml +└── feature/ + └── template.yml +``` + +With the above structure we could have a top-level component that can be used as the +default component. For example, `myorg/rails-rspec` could run all the test profiles together. +However, more specific test profiles could be used separately (`myorg/rails-rspec/integration`). + +**Additional notes:** +- Nesting of components is not permitted. + +## Pipeline Components + +A pipeline component is a reusable single pipeline configuration unit. You can use them to compose a part or an entire pipeline configuration. It can optionally take [input parameters](#specinput-parameters). + +### Writing a pipeline component + +The YAML contains two documents. The first document representes the specifications, while the second document represents +the content. + +#### `spec:input` parameters + +`spec:` allows you to define input arguments for a CI configuration. They must be specified according to the following schema: + +``` +spec: + inputs: + website: # by default all declared inputs are mandatory. + environment: + default: test # apply default if not provided. This makes the input optional. + flags: + default: null # make an input entirely optional with no value by default. + test_run: + options: # a choice must be made from the list since there is no default value. + - unit + - integration + - system +--- +# content of the component +my-job: + script: echo +``` + + +Input parameters are used as follows: + +``` +include: + - component: gitlab.com/org/my-component@1.0 + with: + website: ${MY_WEBSITE} # variables expansion + test_run: system + environment: $[[ inputs.environment ]] # interpolation of upstream inputs +``` + +Using interpolation: + +``` +spec: + inputs: + environment: + options: [test, staging, production] +--- +"run-tests-$[[ inputs.environment ]]": + script: ./run-test + +scan-website: + script: ./scan-website $[[ inputs.environment ]] + rules: + - if: $[[ inputs.environment ]] == 'staging' + - if: $[[ inputs.environment ]] == 'production' + +``` + +**Limits:** +- An interpolation block should not be larger than 1 kilobyte. +- A YAML value with interpolation in it can't be larger than 1 megabyte. +- A YAML configuration can't consist of more than half million entries. + +### Using a pipeline component + +A pipeline component is identified by a unique address in the form `/@` containing: + +**1. FQDN (Fully Qualified Domain Name).** +Currently, we only support addresses that point to the same GitLab instance, meaning that the FQDN matches the GitLab +host. + +**2. The component path** +The component YAML file should be `template.yml`, following the filename convention `.yml`, where `template` is the type of file used for components used under the `include` keyword. + + +The component path can be: + +- A path to a project: `gitlab.com/gitlab-org/dast`. The default component is processed. +- A path to an explicit component: `gitlab.com/gitlab-org/dast/api-scan`. In this case the explicit api-scan component is processed. +- A relative path to a local directory: `./path/to/component`. This path must contain the component YAML that defines the component. + The path must start with `./` or `../`. + +Relative local paths are a abbreviated form of the full component address, meaning that `./path/to/component` called from +a file `mydir/file.yml` in `gitlab-org/dast` project would be expanded to: + +`gitlab.com/gitlab-org/dast/mydir/path/to/component@` + +Based on the context where the component is used we fetch the correct YAML file. For example: + +- If we are including a component `gitlab.com/gitlab-org/dast@1.0` we expect a YAML file named `template.yml` in the +root directory of `gitlab-org/dast` repository. +- If we are including a component `gitlab.com/gitlab-org/dast/api-scan@1.0` we expect a YAML file named `template.yml` inside a +directory `api-scan` of the `gitlab-org/dast` repository. +The directly identified by the component path must contain at least: + +**3. A specific version** +The version of the component can be (in order of highest priority first): + +1. A commit SHA - For example: `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` +2. A tag - For example: `gitlab.com/gitlab-org/dast@1.0` +3. A special moving target version that points to the most recent released tag - For example: `gitlab.com/gitlab-org/dast@~latest` +4. A branch name - For example: `gitlab.com/gitlab-org/dast@master` + +**Additional notes:** +- If a tag and branch exist with the same name, the tag takes precedence over the branch. +- If a tag is named `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, a commit SHA (if exists) takes precedence over the tag. +- A component must be defined in a Git repository. +- When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches +the commit SHA of the current pipeline context. + +#### Using input parameters for existing `include:` syntax + +You may use the `with:` keyword with any included configuration file that uses `spec:inputs` to define arguments. + +```yaml +include: + - component: gitlab.com/org/my-component@1.0 + with: + stage: test + - local: path/to/file.yml + with: + stage: test + - project: org/another + file: .gitlab-ci.yml + with: + stage: test + - remote: http://example.com/ci/config + with: + stage: test + - template: Auto-DevOps.gitlab-ci.yml + with: + stage: test +``` + +The configuration file included must specify the inputs by defining the specification section in their YAML: + +```yaml +spec: + inputs: + stage: +--- +# rest of configuration +``` + +#### Input parameters for pipelines + +Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. + +To trigger a pipeline via the API. + +`POST /projects/:id/pipelines/trigger` with: `{ inputs: { provider: 'aws' } }` + +To trigger a pipeline via the `trigger` syntax: + +```yaml +deploy-app: + trigger: + project: org/deployer + with: + provider: aws + deploy_environment: staging +``` + +Fully leveraging the `inputs` specifications: + +```yaml +spec: + inputs: + concurrency: + default: 10 + provider: + description: Deployment provider + deploy_environment: + options: + - staging + - canary + - production + default: staging + description: Deployment environment # optional: render as input label. +--- +# rest of the pipeline config +``` + +- The `provider` input can enforce `required` in the form validation. +- The `description` can be rendered as input label. +- `staging` will be the default `deploy_enviroment` in the UI + +### Releasing + +The versions that should be displayed for the resource should be the project [releases](). +Creating project releases is an official act of versioning a resource. -- GitLab From 482a8317a418c7a047032c14014dceaa26cad832 Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Wed, 12 Apr 2023 17:21:53 +0200 Subject: [PATCH 02/13] Removes mention from spec:inputs --- doc/ci/components/index.md | 161 +++++-------------------------------- 1 file changed, 20 insertions(+), 141 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 0c071df4073e4e..55cfe211fe931b 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -12,19 +12,20 @@ type: reference This feature is in Alpha and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. Tell us about your use case by leaving comments in the epic. -NOTE: This feature is in Alpha. In future releases, to accomodate new features, it might change without notice. +NOTE: This feature is in Alpha. In future releases, to accomodate new features, it might change without notice. ---- The CI/CD Catalog is an index of resources that you can leverage in CI/CD. It contains a list of components repositories that can be discovered and used in pipelines. -See an example of [development workflow]() for a component repository. +See an example of [development workflow](#writing-a-pipeline-component) for a component repository. ## Components Repository -A components repository is a GitLab project/repository that exclusively hosts one or more pipeline components. +A components repository is a GitLab project/repository that exclusively hosts one or more [pipeline +components](#pipeline-components). -**Structure of a components repository** +### Structure of a components repository A components repository can host one or more components. The following directory structure would support 1 component in a repository with the name `rails-rspec`: @@ -79,73 +80,33 @@ However, more specific test profiles could be used separately (`myorg/rails-rspe **Additional notes:** - Nesting of components is not permitted. -## Pipeline Components - -A pipeline component is a reusable single pipeline configuration unit. You can use them to compose a part or an entire pipeline configuration. It can optionally take [input parameters](#specinput-parameters). - -### Writing a pipeline component - -The YAML contains two documents. The first document representes the specifications, while the second document represents -the content. -#### `spec:input` parameters +### Marking a project as a components repository -`spec:` allows you to define input arguments for a CI configuration. They must be specified according to the following schema: +In order to mark a project as a components repository, you must run the following mutation: ``` -spec: - inputs: - website: # by default all declared inputs are mandatory. - environment: - default: test # apply default if not provided. This makes the input optional. - flags: - default: null # make an input entirely optional with no value by default. - test_run: - options: # a choice must be made from the list since there is no default value. - - unit - - integration - - system ---- -# content of the component -my-job: - script: echo +mutation { + catalogResourcesCreate(input: { projectPath: "path-to-project"}) { + errors + } +} ``` +Note: A UI option will be available soon. Please refer to the [epic](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track the progress. -Input parameters are used as follows: - -``` -include: - - component: gitlab.com/org/my-component@1.0 - with: - website: ${MY_WEBSITE} # variables expansion - test_run: system - environment: $[[ inputs.environment ]] # interpolation of upstream inputs -``` +## Pipeline Components -Using interpolation: +A pipeline component is a reusable single pipeline configuration unit. You can use them to compose a part or an entire pipeline configuration. It can optionally take [input parameters](#). -``` -spec: - inputs: - environment: - options: [test, staging, production] ---- -"run-tests-$[[ inputs.environment ]]": - script: ./run-test +### Writing a pipeline component -scan-website: - script: ./scan-website $[[ inputs.environment ]] - rules: - - if: $[[ inputs.environment ]] == 'staging' - - if: $[[ inputs.environment ]] == 'production' +The YAML contains up to two documents. The optional first document is the header and can be used to define a +specification for the component, while the second document represents the content. -``` +### Releasing a pipeline component -**Limits:** -- An interpolation block should not be larger than 1 kilobyte. -- A YAML value with interpolation in it can't be larger than 1 megabyte. -- A YAML configuration can't consist of more than half million entries. +Currently, the only way to release a component is via the [Releases API](../../api/releases/index.md#create-a-release). ### Using a pipeline component @@ -193,85 +154,3 @@ The version of the component can be (in order of highest priority first): - A component must be defined in a Git repository. - When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches the commit SHA of the current pipeline context. - -#### Using input parameters for existing `include:` syntax - -You may use the `with:` keyword with any included configuration file that uses `spec:inputs` to define arguments. - -```yaml -include: - - component: gitlab.com/org/my-component@1.0 - with: - stage: test - - local: path/to/file.yml - with: - stage: test - - project: org/another - file: .gitlab-ci.yml - with: - stage: test - - remote: http://example.com/ci/config - with: - stage: test - - template: Auto-DevOps.gitlab-ci.yml - with: - stage: test -``` - -The configuration file included must specify the inputs by defining the specification section in their YAML: - -```yaml -spec: - inputs: - stage: ---- -# rest of configuration -``` - -#### Input parameters for pipelines - -Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. - -To trigger a pipeline via the API. - -`POST /projects/:id/pipelines/trigger` with: `{ inputs: { provider: 'aws' } }` - -To trigger a pipeline via the `trigger` syntax: - -```yaml -deploy-app: - trigger: - project: org/deployer - with: - provider: aws - deploy_environment: staging -``` - -Fully leveraging the `inputs` specifications: - -```yaml -spec: - inputs: - concurrency: - default: 10 - provider: - description: Deployment provider - deploy_environment: - options: - - staging - - canary - - production - default: staging - description: Deployment environment # optional: render as input label. ---- -# rest of the pipeline config -``` - -- The `provider` input can enforce `required` in the form validation. -- The `description` can be rendered as input label. -- `staging` will be the default `deploy_enviroment` in the UI - -### Releasing - -The versions that should be displayed for the resource should be the project [releases](). -Creating project releases is an official act of versioning a resource. -- GitLab From cf7b2e610419ae6e0814fc1ab18c2cc938b82e2c Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Fri, 21 Apr 2023 16:58:56 +0900 Subject: [PATCH 03/13] Formatting and structural cleanup --- doc/ci/components/index.md | 185 +++++++++++++++++-------------------- 1 file changed, 87 insertions(+), 98 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 55cfe211fe931b..17df8912575c5a 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -5,152 +5,141 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# CI/CD Catalog (Alpha) +# CI/CD Catalog (Experimental) -> - [Introduced]() in Gitlab 15.11. +> Introduced in GitLab 16.0 as an [experimental feature](../../policy/alpha-beta-support.md). -This feature is in Alpha and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. -Tell us about your use case by leaving comments in the epic. +FLAG: +On self-managed GitLab, by default this feature is not available. +To make it available, ask an administrator to enable the feature flag named `example_flag`. +On GitLab.com, this feature is not available. This feature is not ready for production use. -NOTE: This feature is in Alpha. In future releases, to accomodate new features, it might change without notice. +The CI/CD Catalog is a list of [components repositories](#components-repository), +each containing resources that you can add to your CI/CD pipelines. ----- +This feature is an experimental feature and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) +to track future work. Tell us about your use case by leaving comments in the epic. -The CI/CD Catalog is an index of resources that you can leverage in CI/CD. It contains a list of components repositories that can be discovered and used in pipelines. +## Components Repository -See an example of [development workflow](#writing-a-pipeline-component) for a component repository. +A components repository is a GitLab project with a repository that hosts +one or more [pipeline components](#pipeline-components). -## Components Repository +### File structure -A components repository is a GitLab project/repository that exclusively hosts one or more [pipeline -components](#pipeline-components). +Components repositories must follow a mandatory file structure, containing: -### Structure of a components repository -A components repository can host one or more components. +- `template.yml`: The component configuration, one file per component. If there is + only one component, this file can be in the root of the project. If there are multiple + components, each file must be in a separate subdirectory. +- `README.md`: A Markdown file explaining the details of the components. -The following directory structure would support 1 component in a repository with the name `rails-rspec`: +For example, if the project is on GitLab.com, named `my-component`, and in a personal +namespace named `my-username`: -```yaml -├── template.yml -├── README.md -└── .gitlab-ci.yml -``` +- Containing a single component and a simple pipeline to test the component, then + the file structure might be: -- The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. -- The component is identified by the path `gitlab.com/gitlab-org/rails-rspec`, which is the project path. -- We expect a `template.yml` file and `README.md` to be located in the root directory of the repository. + ```plaintext + ├── template.yml + ├── README.md + └── .gitlab-ci.yml + ``` -The following directory structure would support multiple components per repository: + This component is referenced with the path `gitlab.com/my-username/my-component`. +- Containing one default component and multiple sub-components, then the file structure + might be: -```yaml -├── .gitlab-ci.yml -├── README.md -├── unit/ -│ └── template.yml -├── integration/ -│ └── template.yml -└── feature/ - └── template.yml -``` + ```plaintext + ├── template.yml + ├── README.md + ├── .gitlab-ci.yml + ├── unit/ + │ └── template.yml + └── integration/ + └── template.yml + ``` -Each of these components are identified by their path `gitlab.com/gitlab-org/rails-rspec/unit`, `gitlab.com/myorg/rails-rspec/integration` -and `gitlab.com/myorg/rails-rspec/feature`. - -This directory structure could also support both strategies: - -```yaml -. -├── template.yml -├── README.md -├── LICENSE -├── .gitlab-ci.yml -├── unit/ -│ └── template.yml -├── integration/ -│ └── template.yml -└── feature/ - └── template.yml -``` + These components are identified by these paths: + + - `gitlab.com/my-username/my-component` + - `gitlab.com/my-username/my-component/unit` + - `gitlab.com/my-username/my-component/integration` -With the above structure we could have a top-level component that can be used as the -default component. For example, `myorg/rails-rspec` could run all the test profiles together. -However, more specific test profiles could be used separately (`myorg/rails-rspec/integration`). +It is possible to have a components repository with no default component, by having +no `template.yml` in the root directory. **Additional notes:** -- Nesting of components is not permitted. +- Nesting of components is not permitted. -### Marking a project as a components repository +### Mark a project as a components repository -In order to mark a project as a components repository, you must run the following mutation: +To mark a project as a components repository, you must run the following [graphQL](../../api/graphql/index.md) +mutation: -``` +```graphql mutation { catalogResourcesCreate(input: { projectPath: "path-to-project"}) { - errors + errors } } ``` -Note: A UI option will be available soon. Please refer to the [epic](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track the progress. - ## Pipeline Components -A pipeline component is a reusable single pipeline configuration unit. You can use them to compose a part or an entire pipeline configuration. It can optionally take [input parameters](#). +A pipeline component is a reusable single pipeline configuration unit. You can use +them to compose a part or an entire pipeline configuration. It can optionally take +[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). -### Writing a pipeline component +### Create a component -The YAML contains up to two documents. The optional first document is the header and can be used to define a -specification for the component, while the second document represents the content. +To add a pipeline component in your components repository, create a `template.yml` file. +The file contains one or two parts: -### Releasing a pipeline component +- Optional. A header that defines the component's [specifications](../yaml/includes.md#define-input-parameters-with-specinputs). +- The component's [YAML configuration](../yaml/index.md) -Currently, the only way to release a component is via the [Releases API](../../api/releases/index.md#create-a-release). +### Release a component -### Using a pipeline component +To release a component, use the [Releases API](../../api/releases/index.md#create-a-release). -A pipeline component is identified by a unique address in the form `/@` containing: +### Use a component -**1. FQDN (Fully Qualified Domain Name).** -Currently, we only support addresses that point to the same GitLab instance, meaning that the FQDN matches the GitLab -host. +A pipeline component is identified by a unique address in the form `/@` +containing: -**2. The component path** -The component YAML file should be `template.yml`, following the filename convention `.yml`, where `template` is the type of file used for components used under the `include` keyword. +- A fully qualified domain name (FQDN): The FQDN must match the GitLab host. +- A specific version: The version of the component can be (in order of highest priority first): + - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. + - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. + - `~latest`, which is special version that always points to the most recent released tag. + For example `gitlab.com/gitlab-org/dast@~latest`. + - A branch name , for example: `gitlab.com/gitlab-org/dast@master` + +- **A component path**: The component YAML file should be `template.yml`, following + the filename convention `.yml`, where `template` is the type of file used for + components used under the `include` keyword. The component path can be: - A path to a project: `gitlab.com/gitlab-org/dast`. The default component is processed. -- A path to an explicit component: `gitlab.com/gitlab-org/dast/api-scan`. In this case the explicit api-scan component is processed. +- A path to an explicit component: `gitlab.com/gitlab-org/dast/api-scan`. In this case the explicit `api-scan` component is processed. - A relative path to a local directory: `./path/to/component`. This path must contain the component YAML that defines the component. The path must start with `./` or `../`. -Relative local paths are a abbreviated form of the full component address, meaning that `./path/to/component` called from -a file `mydir/file.yml` in `gitlab-org/dast` project would be expanded to: - -`gitlab.com/gitlab-org/dast/mydir/path/to/component@` +Relative local paths are an abbreviated form of the full component address, +meaning that `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` +project would expand to: -Based on the context where the component is used we fetch the correct YAML file. For example: - -- If we are including a component `gitlab.com/gitlab-org/dast@1.0` we expect a YAML file named `template.yml` in the -root directory of `gitlab-org/dast` repository. -- If we are including a component `gitlab.com/gitlab-org/dast/api-scan@1.0` we expect a YAML file named `template.yml` inside a -directory `api-scan` of the `gitlab-org/dast` repository. -The directly identified by the component path must contain at least: - -**3. A specific version** -The version of the component can be (in order of highest priority first): - -1. A commit SHA - For example: `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` -2. A tag - For example: `gitlab.com/gitlab-org/dast@1.0` -3. A special moving target version that points to the most recent released tag - For example: `gitlab.com/gitlab-org/dast@~latest` -4. A branch name - For example: `gitlab.com/gitlab-org/dast@master` +- `gitlab.com/gitlab-org/dast/mydir/path/to/component@` **Additional notes:** + - If a tag and branch exist with the same name, the tag takes precedence over the branch. -- If a tag is named `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, a commit SHA (if exists) takes precedence over the tag. -- A component must be defined in a Git repository. -- When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches -the commit SHA of the current pipeline context. +- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, + the commit SHA takes precedence over the tag. +- When referencing a component by using a local path like `./path/to/component`, the version is + the commit SHA of the current pipeline context. -- GitLab From 8594e7b7024c4b350ad85853344506fabc882370 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Tue, 2 May 2023 14:07:46 +0000 Subject: [PATCH 04/13] Formatting suggestions --- doc/ci/components/index.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 17df8912575c5a..1dc7fd21888656 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -123,18 +123,16 @@ containing: the filename convention `.yml`, where `template` is the type of file used for components used under the `include` keyword. -The component path can be: + For example, for a component repository located at `gitlab.com/gitlab-org/dast`, + the path could be: -- A path to a project: `gitlab.com/gitlab-org/dast`. The default component is processed. -- A path to an explicit component: `gitlab.com/gitlab-org/dast/api-scan`. In this case the explicit `api-scan` component is processed. -- A relative path to a local directory: `./path/to/component`. This path must contain the component YAML that defines the component. - The path must start with `./` or `../`. + - The path to the project: `gitlab-org/dast`. The default component in the `dast` project is used. + - The path to a sub-component: `gitlab-org/dast/api-scan`. The `api-scan` sub-component is used. + - The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component. + The path must start with `./` or `../`. -Relative local paths are an abbreviated form of the full component address, -meaning that `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` -project would expand to: - -- `gitlab.com/gitlab-org/dast/mydir/path/to/component@` + For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` + project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` **Additional notes:** -- GitLab From 01cb2b59e50a88c4724b823ea17edda560faab75 Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Wed, 3 May 2023 19:17:54 +0200 Subject: [PATCH 05/13] Updates with feedback --- doc/ci/components/index.md | 107 ++++++++++++++++++++++++++++++------- 1 file changed, 87 insertions(+), 20 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 1dc7fd21888656..dcf0a29fa12a77 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -11,7 +11,8 @@ type: reference FLAG: On self-managed GitLab, by default this feature is not available. -To make it available, ask an administrator to enable the feature flag named `example_flag`. +To make it available, ask an administrator to enable the feature flags named `ci_include_components` and +`ci_private_catalog_beta`. On GitLab.com, this feature is not available. This feature is not ready for production use. The CI/CD Catalog is a list of [components repositories](#components-repository), @@ -25,7 +26,21 @@ to track future work. Tell us about your use case by leaving comments in the epi A components repository is a GitLab project with a repository that hosts one or more [pipeline components](#pipeline-components). -### File structure +### Mark a project as a components repository + +To mark a project as a components repository, you must run the following [graphQL](../../api/graphql/index.md) +mutation: + +```graphql +mutation { + catalogResourcesCreate(input: { projectPath: "path-to-project"}) { + errors + } +} +``` +### Structure of a components repository + +A components repository can host one or more components. The author can decide whether to define a single component per repository or include multiple cohesive components in the same repository. Components repositories must follow a mandatory file structure, containing: @@ -74,18 +89,6 @@ no `template.yml` in the root directory. - Nesting of components is not permitted. -### Mark a project as a components repository - -To mark a project as a components repository, you must run the following [graphQL](../../api/graphql/index.md) -mutation: - -```graphql -mutation { - catalogResourcesCreate(input: { projectPath: "path-to-project"}) { - errors - } -} -``` ## Pipeline Components @@ -95,15 +98,79 @@ them to compose a part or an entire pipeline configuration. It can optionally ta ### Create a component -To add a pipeline component in your components repository, create a `template.yml` file. -The file contains one or two parts: +1. Create a new project and add a `README.md` file. + +2. Create a component inside the project. + +If you intend to have only one component in the repository, you can define it in the root directory. +Otherwise, create a directory for the component. + +``` +spec: + inputs: + stage: + default: test +--- +.component-default-job: + image: busybox + stage: $[[ inputs.stage ]] + +component-job-1: + extends: .component-default-job + script: echo job 1 + +component-job-2: + extends: .component-default-job + script: echo job 2 +``` + +3. Test changes in CI by creating a `.gitlab-ci.yml` in the root directory. + +``` +## +# This configuration expects an access token with read-only access to the API +# to be saved as in a masked CI/CD variable named 'API_TOKEN' + +include: + # Leverage predefined variables to refer to the current project and SHA + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + +stages: [test, release] + +# Expect all `component-job-*` jobs are added +ensure-jobs-added: + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job-"))) | length'` + if [ "$count" != "2" ]; then + exit 1 + fi + +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` + +4. Run a new pipeline for the `main` branch by pushing a commit or manually running it. + +5. Create a [tag](https://docs.gitlab.com/ee/user/project/repository/tags/index.html#create-a-tag). -- Optional. A header that defines the component's [specifications](../yaml/includes.md#define-input-parameters-with-specinputs). -- The component's [YAML configuration](../yaml/index.md) +Wait for the pipeline to be green. When the `create-release` job finishes we should see the new release available in +the Releases menu. -### Release a component +6. Publish the repository to the catalog. -To release a component, use the [Releases API](../../api/releases/index.md#create-a-release). +Publishing a components repository makes it a catalog resource. ### Use a component -- GitLab From e62cad1b6d25b05f88f4487483d5c4d092552e2d Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Thu, 4 May 2023 17:50:06 +0200 Subject: [PATCH 06/13] Adds a quickstart guide at the top --- doc/ci/components/index.md | 181 ++++++++++++++++--------------------- 1 file changed, 78 insertions(+), 103 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index dcf0a29fa12a77..f917c28dee8318 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -5,14 +5,77 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# CI/CD Catalog (Experimental) +# Quickstart for pipeline components + +A pipeline component is a reusable single pipeline configuration unit. You can use +them to compose a part or an entire pipeline configuration. It can optionally take +[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). + +### Create your first component + +1. Create a new project and add a `README.md` file. + +2. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. + +``` +spec: + inputs: + stage: + default: test +--- +component-job: + script: echo job 1 + stage: $[[ inputs.stage ]] +``` + +3. Test changes in CI by creating a `.gitlab-ci.yml` in the root directory. + +``` +include: + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + +stages: test + +# Expect `component-job` is added +ensure-job-added: + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` + if [ "$count" != "1" ]; then + exit 1 + fi + +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` + +4. Run a new pipeline for the `main` branch by pushing a commit or manually running it. + +5. Create a [tag](https://docs.gitlab.com/ee/user/project/repository/tags/index.html#create-a-tag). + +Wait for the pipeline to be green. When the `create-release` job finishes we should see the new release available in +the [Releases](https://docs.gitlab.com/ee/user/project/releases/#releases) menu. + +You are now ready to publish the repository to the [catalog](cicd-catalog-experimental). + +## CI/CD Catalog (Experimental) > Introduced in GitLab 16.0 as an [experimental feature](../../policy/alpha-beta-support.md). FLAG: On self-managed GitLab, by default this feature is not available. -To make it available, ask an administrator to enable the feature flags named `ci_include_components` and -`ci_private_catalog_beta`. +To make it available, ask an administrator to enable the feature flag named `ci_namespace_catalog_experimental`. On GitLab.com, this feature is not available. This feature is not ready for production use. The CI/CD Catalog is a list of [components repositories](#components-repository), @@ -40,7 +103,7 @@ mutation { ``` ### Structure of a components repository -A components repository can host one or more components. The author can decide whether to define a single component per repository or include multiple cohesive components in the same repository. +A components repository can host one or more components. Components repositories must follow a mandatory file structure, containing: @@ -89,117 +152,25 @@ no `template.yml` in the root directory. - Nesting of components is not permitted. - -## Pipeline Components - -A pipeline component is a reusable single pipeline configuration unit. You can use -them to compose a part or an entire pipeline configuration. It can optionally take -[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). - -### Create a component - -1. Create a new project and add a `README.md` file. - -2. Create a component inside the project. - -If you intend to have only one component in the repository, you can define it in the root directory. -Otherwise, create a directory for the component. - -``` -spec: - inputs: - stage: - default: test ---- -.component-default-job: - image: busybox - stage: $[[ inputs.stage ]] - -component-job-1: - extends: .component-default-job - script: echo job 1 - -component-job-2: - extends: .component-default-job - script: echo job 2 -``` - -3. Test changes in CI by creating a `.gitlab-ci.yml` in the root directory. - -``` -## -# This configuration expects an access token with read-only access to the API -# to be saved as in a masked CI/CD variable named 'API_TOKEN' - -include: - # Leverage predefined variables to refer to the current project and SHA - - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA - -stages: [test, release] - -# Expect all `component-job-*` jobs are added -ensure-jobs-added: - image: badouralix/curl-jq - script: - - | - route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" - count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job-"))) | length'` - if [ "$count" != "2" ]; then - exit 1 - fi - -# If we are tagging a release with a specific convention ("v" + number) and all -# previous checks succeeded, we proceed with creating a release automatically. -create-release: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG =~ /^v\d+/ - script: echo "Creating release $CI_COMMIT_TAG" - release: - tag_name: $CI_COMMIT_TAG - description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" -``` - -4. Run a new pipeline for the `main` branch by pushing a commit or manually running it. - -5. Create a [tag](https://docs.gitlab.com/ee/user/project/repository/tags/index.html#create-a-tag). - -Wait for the pipeline to be green. When the `create-release` job finishes we should see the new release available in -the Releases menu. - -6. Publish the repository to the catalog. - -Publishing a components repository makes it a catalog resource. - -### Use a component +### Include a component in a CI Configuration A pipeline component is identified by a unique address in the form `/@` containing: -- A fully qualified domain name (FQDN): The FQDN must match the GitLab host. -- A specific version: The version of the component can be (in order of highest priority first): - +- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. +- **A specific version**: The version of the component can be (in order of highest priority first): - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - `~latest`, which is special version that always points to the most recent released tag. For example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name , for example: `gitlab.com/gitlab-org/dast@master` - -- **A component path**: The component YAML file should be `template.yml`, following - the filename convention `.yml`, where `template` is the type of file used for - components used under the `include` keyword. + - A branch name, for example: `gitlab.com/gitlab-org/dast@master` +- **A component path**: The component YAML file should be `template.yml` - For example, for a component repository located at `gitlab.com/gitlab-org/dast`, - the path could be: +For example, for a component repository located at `gitlab.com/gitlab-org/dast`, the path could be: - The path to the project: `gitlab-org/dast`. The default component in the `dast` project is used. - The path to a sub-component: `gitlab-org/dast/api-scan`. The `api-scan` sub-component is used. - - The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component. - The path must start with `./` or `../`. - - For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` - project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` + - The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component.The path must start with `./` or `../`. For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` **Additional notes:** @@ -208,3 +179,7 @@ containing: the commit SHA takes precedence over the tag. - When referencing a component by using a local path like `./path/to/component`, the version is the commit SHA of the current pipeline context. + +### Releasing a component + +Component repositories are released using the [`release`](https://docs.gitlab.com/ee/ci/yaml/#release) keyword. -- GitLab From d01dae633a3a9b015d5777dd4ecb0139b54e7a4c Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Tue, 9 May 2023 13:32:25 +0000 Subject: [PATCH 07/13] Technical writing feedback --- doc/ci/components/index.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index f917c28dee8318..26f65f42f52b1d 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -1,6 +1,6 @@ --- stage: Verify -group: Pipeline Composition +group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments type: reference --- @@ -8,16 +8,16 @@ type: reference # Quickstart for pipeline components A pipeline component is a reusable single pipeline configuration unit. You can use -them to compose a part or an entire pipeline configuration. It can optionally take +them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). -### Create your first component +### Create a component -1. Create a new project and add a `README.md` file. +1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. 2. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. -``` +```yaml spec: inputs: stage: @@ -64,8 +64,8 @@ create-release: 5. Create a [tag](https://docs.gitlab.com/ee/user/project/repository/tags/index.html#create-a-tag). -Wait for the pipeline to be green. When the `create-release` job finishes we should see the new release available in -the [Releases](https://docs.gitlab.com/ee/user/project/releases/#releases) menu. +After the `create-release` job finishes, the new release is available on +the [**Releases**](https://docs.gitlab.com/ee/user/project/releases/#releases) page. You are now ready to publish the repository to the [catalog](cicd-catalog-experimental). -- GitLab From 79ebae88a5eff4e5a7a5e4f5e1e880344360ec16 Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Tue, 9 May 2023 16:28:10 +0200 Subject: [PATCH 08/13] Moves quickstart to the bottom, makes testing section --- doc/ci/components/index.md | 143 +++++++++++++++++++------------------ 1 file changed, 74 insertions(+), 69 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 26f65f42f52b1d..7876c05838c492 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -5,71 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Quickstart for pipeline components - -A pipeline component is a reusable single pipeline configuration unit. You can use -them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take -[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). - -### Create a component - -1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. - -2. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. - -```yaml -spec: - inputs: - stage: - default: test ---- -component-job: - script: echo job 1 - stage: $[[ inputs.stage ]] -``` - -3. Test changes in CI by creating a `.gitlab-ci.yml` in the root directory. - -``` -include: - - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA - -stages: test - -# Expect `component-job` is added -ensure-job-added: - image: badouralix/curl-jq - script: - - | - route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" - count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` - if [ "$count" != "1" ]; then - exit 1 - fi - -# If we are tagging a release with a specific convention ("v" + number) and all -# previous checks succeeded, we proceed with creating a release automatically. -create-release: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG =~ /^v\d+/ - script: echo "Creating release $CI_COMMIT_TAG" - release: - tag_name: $CI_COMMIT_TAG - description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" -``` - -4. Run a new pipeline for the `main` branch by pushing a commit or manually running it. - -5. Create a [tag](https://docs.gitlab.com/ee/user/project/repository/tags/index.html#create-a-tag). - -After the `create-release` job finishes, the new release is available on -the [**Releases**](https://docs.gitlab.com/ee/user/project/releases/#releases) page. - -You are now ready to publish the repository to the [catalog](cicd-catalog-experimental). - -## CI/CD Catalog (Experimental) +# CI/CD Catalog (Experimental) > Introduced in GitLab 16.0 as an [experimental feature](../../policy/alpha-beta-support.md). @@ -101,6 +37,7 @@ mutation { } } ``` + ### Structure of a components repository A components repository can host one or more components. @@ -150,7 +87,37 @@ no `template.yml` in the root directory. **Additional notes:** -- Nesting of components is not permitted. +- Nesting of components is not permitted. Example: + + ```plaintext + ├── unit/ + │ └── template.yml + │ └── another_folder/ + │ └── nested_template.yml + ``` + +## Pipeline Components + +A pipeline component is a reusable single pipeline configuration unit. You can use +them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take +[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). + +### Create a component + +1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. + +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. + +```yaml +spec: + inputs: + stage: + default: test +--- +component-job: + script: echo job 1 + stage: $[[ inputs.stage ]] +``` ### Include a component in a CI Configuration @@ -168,9 +135,9 @@ containing: For example, for a component repository located at `gitlab.com/gitlab-org/dast`, the path could be: - - The path to the project: `gitlab-org/dast`. The default component in the `dast` project is used. - - The path to a sub-component: `gitlab-org/dast/api-scan`. The `api-scan` sub-component is used. - - The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component.The path must start with `./` or `../`. For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` +- The path to the project: `gitlab-org/dast`. The default component in the `dast` project is used. +- The path to a sub-component: `gitlab-org/dast/api-scan`. The `api-scan` sub-component is used. +- The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component. The path must start with `./` or `../`. For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` **Additional notes:** @@ -180,6 +147,44 @@ For example, for a component repository located at `gitlab.com/gitlab-org/dast`, - When referencing a component by using a local path like `./path/to/component`, the version is the commit SHA of the current pipeline context. +### Testing a component + +We strongly recommend testing the components as part of the development workflow. Testing changes in CI can be done by creating a `.gitlab-ci.yml` in the root directory. + +Example: + +```yaml +include: + - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + +stages: test + +# Expect `component-job` is added +ensure-job-added: + image: badouralix/curl-jq + script: + - | + route="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_ID/jobs" + count=`curl --silent --header "PRIVATE-TOKEN: $API_TOKEN" $route | jq 'map(select(.name | contains("component-job"))) | length'` + if [ "$count" != "1" ]; then + exit 1 + fi + +# If we are tagging a release with a specific convention ("v" + number) and all +# previous checks succeeded, we proceed with creating a release automatically. +create-release: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG =~ /^v\d+/ + script: echo "Creating release $CI_COMMIT_TAG" + release: + tag_name: $CI_COMMIT_TAG + description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" +``` + +A new pipeline for the `main` branch can now be triggered by pushing a commit or manually running it. + ### Releasing a component Component repositories are released using the [`release`](https://docs.gitlab.com/ee/ci/yaml/#release) keyword. -- GitLab From 274dd33d667a9392b2ba42e662ca19ada662c1fe Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Tue, 9 May 2023 17:13:41 +0000 Subject: [PATCH 09/13] Applies Fabio's suggestions --- doc/ci/components/index.md | 44 ++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 7876c05838c492..fef221c6a872b2 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -25,9 +25,14 @@ to track future work. Tell us about your use case by leaving comments in the epi A components repository is a GitLab project with a repository that hosts one or more [pipeline components](#pipeline-components). -### Mark a project as a components repository +### Mark the project as a catalog resource -To mark a project as a components repository, you must run the following [graphQL](../../api/graphql/index.md) +The components we created so far in this repository can be [used](#include-a-component-in-a-ci-configuration) to build pipelines in other projects. + +However, this repository is not discoverable. By marking this project as a catalog resource we allow it to be visible in the CI Catalog +so other users can discover it. + +To mark a project as a catalog resource, you must run the following [graphQL](../../api/graphql/index.md) mutation: ```graphql @@ -47,7 +52,7 @@ Components repositories must follow a mandatory file structure, containing: - `template.yml`: The component configuration, one file per component. If there is only one component, this file can be in the root of the project. If there are multiple components, each file must be in a separate subdirectory. -- `README.md`: A Markdown file explaining the details of the components. +- `README.md`: A documentation file explaining the details of the all the components in the repository. For example, if the project is on GitLab.com, named `my-component`, and in a personal namespace named `my-username`: @@ -61,7 +66,7 @@ namespace named `my-username`: └── .gitlab-ci.yml ``` - This component is referenced with the path `gitlab.com/my-username/my-component`. + This component is referenced with the path `gitlab.com/my-username/my-component@`. - Containing one default component and multiple sub-components, then the file structure might be: @@ -130,36 +135,38 @@ containing: - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - `~latest`, which is special version that always points to the most recent released tag. For example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example: `gitlab.com/gitlab-org/dast@master` -- **A component path**: The component YAML file should be `template.yml` + - A branch name, for example: `gitlab.com/gitlab-org/dast@main` +- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. -For example, for a component repository located at `gitlab.com/gitlab-org/dast`, the path could be: +For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: -- The path to the project: `gitlab-org/dast`. The default component in the `dast` project is used. -- The path to a sub-component: `gitlab-org/dast/api-scan`. The `api-scan` sub-component is used. -- The relative path to a local directory: `./path/to/component`. This path must contain the `template.yml` that defines the component. The path must start with `./` or `../`. For example, `./path/to/component` called from a file `mydir/file.yml` in `gitlab-org/dast` project would expand to `gitlab.com/gitlab-org/dast/mydir/path/to/component@` +- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. +- The path `gitalb.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. **Additional notes:** - If a tag and branch exist with the same name, the tag takes precedence over the branch. - If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, the commit SHA takes precedence over the tag. -- When referencing a component by using a local path like `./path/to/component`, the version is - the commit SHA of the current pipeline context. ### Testing a component -We strongly recommend testing the components as part of the development workflow. Testing changes in CI can be done by creating a `.gitlab-ci.yml` in the root directory. +We strongly recommend testing the components as part of the development workflow to ensure that quality maintains high standards. + +Testing changes in CI can be done, like any other project, by creating a `.gitlab-ci.yml` in the root directory. Example: ```yaml include: + # include the component located in the current project from the current SHA - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA stages: test -# Expect `component-job` is added +# Expect `component-job` is added. +# This is an example of testing that the included component works as expected. +# You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. ensure-job-added: image: badouralix/curl-jq script: @@ -183,8 +190,13 @@ create-release: description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" ``` -A new pipeline for the `main` branch can now be triggered by pushing a commit or manually running it. +By committing and pushing the changes a new pipeline is created which tests that the component can be used correctly. ### Releasing a component -Component repositories are released using the [`release`](https://docs.gitlab.com/ee/ci/yaml/#release) keyword. +Component repositories are released using the [`release`](https://docs.gitlab.com/ee/ci/yaml/#release) keyword within a CI pipeline. + +Like in the [example above](#testing-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. + +All released versions of the components repository are displayed in the CI catalog page for the given resource, providing users +with information about official releases. -- GitLab From e2b981a0d6087ce5b6230d19ce1363d67c8aa4d1 Mon Sep 17 00:00:00 2001 From: Laura Montemayor Date: Wed, 10 May 2023 12:59:05 +0200 Subject: [PATCH 10/13] Updates layout --- doc/ci/components/index.md | 142 ++++++++++++++++++------------------- 1 file changed, 68 insertions(+), 74 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index fef221c6a872b2..a09e5abbebaa8d 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# CI/CD Catalog (Experimental) +# CI/CD Components (Experimental) > Introduced in GitLab 16.0 as an [experimental feature](../../policy/alpha-beta-support.md). @@ -14,36 +14,31 @@ On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to enable the feature flag named `ci_namespace_catalog_experimental`. On GitLab.com, this feature is not available. This feature is not ready for production use. -The CI/CD Catalog is a list of [components repositories](#components-repository), -each containing resources that you can add to your CI/CD pipelines. - This feature is an experimental feature and [an epic exists](https://gitlab.com/groups/gitlab-org/-/epics/9897) to track future work. Tell us about your use case by leaving comments in the epic. ## Components Repository -A components repository is a GitLab project with a repository that hosts -one or more [pipeline components](#pipeline-components). - -### Mark the project as a catalog resource +A components repository is a GitLab project with a repository that hosts one or more pipeline components. A pipeline component is a reusable single pipeline configuration unit. You can use them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take [input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). -The components we created so far in this repository can be [used](#include-a-component-in-a-ci-configuration) to build pipelines in other projects. +### Create a components repository -However, this repository is not discoverable. By marking this project as a catalog resource we allow it to be visible in the CI Catalog -so other users can discover it. +1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. -To mark a project as a catalog resource, you must run the following [graphQL](../../api/graphql/index.md) -mutation: +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. -```graphql -mutation { - catalogResourcesCreate(input: { projectPath: "path-to-project"}) { - errors - } -} +```yaml +spec: + inputs: + stage: + default: test +--- +component-job: + script: echo job 1 + stage: $[[ inputs.stage ]] ``` -### Structure of a components repository +### Directory structure A components repository can host one or more components. @@ -52,7 +47,7 @@ Components repositories must follow a mandatory file structure, containing: - `template.yml`: The component configuration, one file per component. If there is only one component, this file can be in the root of the project. If there are multiple components, each file must be in a separate subdirectory. -- `README.md`: A documentation file explaining the details of the all the components in the repository. +- `README.md`: A documentation file explaining the details of the all the components in the repository. For example, if the project is on GitLab.com, named `my-component`, and in a personal namespace named `my-username`: @@ -101,55 +96,7 @@ no `template.yml` in the root directory. │ └── nested_template.yml ``` -## Pipeline Components - -A pipeline component is a reusable single pipeline configuration unit. You can use -them to compose an entire pipeline configuration or a small part of a larger pipeline. It can optionally take -[input parameters](../yaml/includes.md#define-input-parameters-with-specinputs). - -### Create a component - -1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. - -1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. - -```yaml -spec: - inputs: - stage: - default: test ---- -component-job: - script: echo job 1 - stage: $[[ inputs.stage ]] -``` - -### Include a component in a CI Configuration - -A pipeline component is identified by a unique address in the form `/@` -containing: - -- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. -- **A specific version**: The version of the component can be (in order of highest priority first): - - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - - `~latest`, which is special version that always points to the most recent released tag. - For example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example: `gitlab.com/gitlab-org/dast@main` -- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. - -For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: - -- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. -- The path `gitalb.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. - -**Additional notes:** - -- If a tag and branch exist with the same name, the tag takes precedence over the branch. -- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, - the commit SHA takes precedence over the tag. - -### Testing a component +### Test a component We strongly recommend testing the components as part of the development workflow to ensure that quality maintains high standards. @@ -192,11 +139,58 @@ create-release: By committing and pushing the changes a new pipeline is created which tests that the component can be used correctly. -### Releasing a component +### Release a component -Component repositories are released using the [`release`](https://docs.gitlab.com/ee/ci/yaml/#release) keyword within a CI pipeline. +Component repositories are released using the [`release`](../yaml/index.md#release) keyword within a CI pipeline. Like in the [example above](#testing-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. -All released versions of the components repository are displayed in the CI catalog page for the given resource, providing users -with information about official releases. +All released versions of the components repository are displayed in the CI catalog page for the given resource, providing users with information about official releases. + +### Use a component in a CI/CD configuration + +A pipeline component is identified by a unique address in the form `/@` +containing: + +- **A fully qualified domain name (FQDN)**: The FQDN must match the GitLab host. +- **A specific version**: The version of the component can be (in order of highest priority first): + - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. + - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. + - `~latest`, which is special version that always points to the most recent released tag. + For example `gitlab.com/gitlab-org/dast@~latest`. + - A branch name, for example: `gitlab.com/gitlab-org/dast@main` +- **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. + +For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: + +- The path `gitlab.com/gitlab-org/dast` tries to load the `template.yml` from the root directory. +- The path `gitalb.com/gitlab-org/dast/api-scan` tries to load the `template.yml` from the `/api-scan` directory. + +**Additional notes:** + +- If a tag and branch exist with the same name, the tag takes precedence over the branch. +- If a tag is named the same as a commit SHA that exists, like `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, + the commit SHA takes precedence over the tag. + +## Components catalog + +The CI/CD Catalog is a list of [components repositories](#components-repository), +each containing resources that you can add to your CI/CD pipelines. + +### Mark the project as a catalog resource + +The components we created so far in this repository can be [used](#include-a-component-in-a-ci-configuration) to build pipelines in other projects. + +However, this repository is not discoverable. By marking this project as a catalog resource we allow it to be visible in the CI Catalog +so other users can discover it. + +To mark a project as a catalog resource, you must run the following [graphQL](../../api/graphql/index.md) +mutation: + +```graphql +mutation { + catalogResourcesCreate(input: { projectPath: "path-to-project"}) { + errors + } +} +``` -- GitLab From a233f22c41f4843a911b7970181a452c98389ed3 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 10 May 2023 11:37:59 +0000 Subject: [PATCH 11/13] Apply 2 suggestion(s) to 1 file(s) --- doc/ci/components/index.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index a09e5abbebaa8d..2c668e8a859c2e 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -108,13 +108,16 @@ Example: include: # include the component located in the current project from the current SHA - component: gitlab.com/$CI_PROJECT_PATH@$CI_COMMIT_SHA + inputs: + stage: build -stages: test +stages: [build, test, release] # Expect `component-job` is added. # This is an example of testing that the included component works as expected. # You can leverage GitLab API endpoints or 3rd party tools to inspect data generated by the component. ensure-job-added: + stage: test image: badouralix/curl-jq script: - | -- GitLab From f4d639e5f06f1d1d49a67966caa2336b2daa2fee Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Wed, 10 May 2023 11:39:03 +0000 Subject: [PATCH 12/13] Apply 2 suggestion(s) to 1 file(s) --- doc/ci/components/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index 2c668e8a859c2e..d73dedf844b3ee 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -146,7 +146,7 @@ By committing and pushing the changes a new pipeline is created which tests that Component repositories are released using the [`release`](../yaml/index.md#release) keyword within a CI pipeline. -Like in the [example above](#testing-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. +Like in the [example above](#test-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. All released versions of the components repository are displayed in the CI catalog page for the given resource, providing users with information about official releases. @@ -182,7 +182,7 @@ each containing resources that you can add to your CI/CD pipelines. ### Mark the project as a catalog resource -The components we created so far in this repository can be [used](#include-a-component-in-a-ci-configuration) to build pipelines in other projects. +Once components are committed and pushed to a repository, they can immediately be [used](#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. However, this repository is not discoverable. By marking this project as a catalog resource we allow it to be visible in the CI Catalog so other users can discover it. -- GitLab From 10157a4cd7f3b63d48bb7c5ba683d41af242a935 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Wed, 10 May 2023 15:07:06 +0000 Subject: [PATCH 13/13] Apply 11 suggestion(s) to 1 file(s) --- doc/ci/components/index.md | 60 ++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 29 deletions(-) diff --git a/doc/ci/components/index.md b/doc/ci/components/index.md index d73dedf844b3ee..82040d5990c89d 100644 --- a/doc/ci/components/index.md +++ b/doc/ci/components/index.md @@ -23,20 +23,22 @@ A components repository is a GitLab project with a repository that hosts one or ### Create a components repository +To create a components repository, you must: + 1. [Create a new project](../../user/project/index.md#create-a-blank-project) with a `README.md` file. -1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. +1. Create a `template.yml` file inside the project's root directory that contains the configuration you want to provide as a component. For example: -```yaml -spec: - inputs: - stage: - default: test ---- -component-job: - script: echo job 1 - stage: $[[ inputs.stage ]] -``` + ```yaml + spec: + inputs: + stage: + default: test + --- + component-job: + script: echo job 1 + stage: $[[ inputs.stage ]] + ``` ### Directory structure @@ -87,22 +89,22 @@ no `template.yml` in the root directory. **Additional notes:** -- Nesting of components is not permitted. Example: +Nesting of components is not possible. For example: - ```plaintext - ├── unit/ - │ └── template.yml - │ └── another_folder/ - │ └── nested_template.yml - ``` +```plaintext +├── unit/ +│ └── template.yml +│ └── another_folder/ +│ └── nested_template.yml +``` ### Test a component -We strongly recommend testing the components as part of the development workflow to ensure that quality maintains high standards. +Testing components as part of the development workflow to ensure that quality maintains high standards is strongly recommended. -Testing changes in CI can be done, like any other project, by creating a `.gitlab-ci.yml` in the root directory. +Testing changes in a CI/CD pipeline can be done, like any other project, by creating a `.gitlab-ci.yml` in the root directory. -Example: +For example: ```yaml include: @@ -140,7 +142,7 @@ create-release: description: "Release $CI_COMMIT_TAG of components repository $CI_PROJECT_PATH" ``` -By committing and pushing the changes a new pipeline is created which tests that the component can be used correctly. +After committing and pushing changes, the pipeline tests the component then releases it if the test passes. ### Release a component @@ -148,7 +150,7 @@ Component repositories are released using the [`release`](../yaml/index.md#relea Like in the [example above](#test-a-component), after all tests pass in a pipeline running for a tag ref, we can release a new version of the components repository. -All released versions of the components repository are displayed in the CI catalog page for the given resource, providing users with information about official releases. +All released versions of the components repository are displayed in the Components Catalog page for the given resource, providing users with information about official releases. ### Use a component in a CI/CD configuration @@ -159,9 +161,9 @@ containing: - **A specific version**: The version of the component can be (in order of highest priority first): - A commit SHA, for example `gitlab.com/gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8`. - A tag. for example: `gitlab.com/gitlab-org/dast@1.0`. - - `~latest`, which is special version that always points to the most recent released tag. - For example `gitlab.com/gitlab-org/dast@~latest`. - - A branch name, for example: `gitlab.com/gitlab-org/dast@main` + - `~latest`, which is a special version that always points to the most recent released tag, + for example `gitlab.com/gitlab-org/dast@~latest`. + - A branch name, for example `gitlab.com/gitlab-org/dast@main`. - **A component path**: Contains the project's full path and the directory where the component YAML file `template.yml` is located. For example, for a component repository located at `gitlab-org/dast` on `gitlab.com`: @@ -182,12 +184,12 @@ each containing resources that you can add to your CI/CD pipelines. ### Mark the project as a catalog resource -Once components are committed and pushed to a repository, they can immediately be [used](#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. +After components are added to a components repository, they can immediately be [used](#use-a-component-in-a-cicd-configuration) to build pipelines in other projects. -However, this repository is not discoverable. By marking this project as a catalog resource we allow it to be visible in the CI Catalog +However, this repository is not discoverable. You must mark this project as a catalog resource to allow it to be visible in the CI Catalog so other users can discover it. -To mark a project as a catalog resource, you must run the following [graphQL](../../api/graphql/index.md) +To mark a project as a catalog resource, run the following [graphQL](../../api/graphql/index.md) mutation: ```graphql -- GitLab