From f4ac3b67d3acb0d703ae64246011497003512804 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Fri, 2 Dec 2022 11:41:35 +0000 Subject: [PATCH 1/6] Specify syntax for component YAML and input parameters --- .../ci_pipeline_components/index.md | 160 ++++++++++++++++++ 1 file changed, 160 insertions(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 08a285acadce18..ded27ad93ad11a 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -173,6 +173,23 @@ A `gitlab-.yml` file: - Can optionally define **output data** that it returns. - Should be **validated statically** (for example: using JSON schema validators). +```yaml +spec: + description: My component description + inputs: + environment: + default: test + website: + required: true + test_run: + required: true + options: + - unit + - integration + - system +content: { ... } +``` + Components that are released in the catalog must have a `README.md` file in the same directory as the metadata YAML file. The `README.md` represents the documentation for the specific component, hence it's recommended even when not releasing versions in the catalog. @@ -274,6 +291,149 @@ NOTE: Any nesting more than 1 level is initially not permitted. This limitation encourages cohesion at project level and keeps complexity low. +## Input parameters `spec:inputs:` parameters + +If the component takes any input parameters they must be specified according to the following schema: + +```yaml +spec: + inputs: + environment: + default: test # apply default if not provided. + website: + required: true # fail if value not provided. + test_run: + required: true # a choice must be made from the list since there is no default value. + options: + - unit + - integration + - system +``` + +When using the component we pass the input parameters as follow: + +```yaml +include: + - component: org/my-component@1.0 + with: + website: http://example.com + test_run: system +``` + +Input parameters are validated as soon as possible: + +1. Read the file `.gitlab-template.yml` inside `org/my-component`. +1. Parse `spec:inputs` and validate the parameters against this schema. +1. If successfully validated, proceed with parsing `content:`. Return an error otherwise. +1. Interpolate input parameters inside the component's `content:`. + +```yaml +spec: + inputs: + environment: + options: [test, staging, production] +content: + run-tests: + script: ./run-test + rules: + - if: $[[ inputs.environment ]] == 'test' + "scan-$[[ inputs.environment ]]-website": + script: ./scan-website ${{ inputs.environment }} + rules: + - if: $[[ inputs.environment ]] == 'staging' + - if: $[[ inputs.environment ]] == 'production' +``` + +- `$[[ inputs.XXX ]]` syntax interpolates the inputs immediately after parsing the `content:`. +- `${{ inputs.XXX }}` syntax interpolates the inputs on the Runner, during runtime. + +> TODO: Do we really need this differentiation? +> ${{ ... }} cannot be used anywhere outside `script:`, `after_script:` and `before_script:`. +> We could instead benefit from interpolating values immediately and save `options[:script]` +> as expanded version. +> +> Runtime interpolation makes sense if the values change in the meantime, but that's not +> the case. + +### Why input parameters and not environment variables? + +Until today we have been leveraging environment variables to pass information around. +For example, we use environment variables to pass information from an upstream pipeline to a +downstream pipeline. + +Using environment variables for passing information to a component is like declaring global +variables in programming languages. The more variables we declare the more we risk variable +conflicts and increase variables scope. + +Input parameters are like local variables that exist inside a specific scope and they don't +leak to the outside. + +This paradigm allows to build more robust and isolated components as well as declare and +enforce contracts. + +### Input parameters for existing `include:` syntax + +Because we are adding input parameters to components used via `include:component` we have an opportunity to +extend it to other `include:` types support inputs via `with:` syntax: + +```yaml +include: + - component: org/my-component@1.0 + with: + foo: bar + - local: path/to/file.yml + with: + foo: bar + - project: org/another + file: .gitlab-ci.yml + with: + foo: bar +``` + +> TODO: what about `template` and `remote`? + +### Input parameters for pipelines + +Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. + +Today we have different use cases where using explicit input parameters would be beneficial: + +1. `Run Pipeline` UI form. + - **Problem today**: We are using top-level variables with `variables:*:description` to surface environment variables to the UI. + The problem with this is the mix of responsibilities as well as the jump in precedence that a variable gets (from job variable to pipeline variable). + Building validation and features on top of this solution is challenging and complex. +1. Trigger a pipeline via API. For example `POST /projects/:id/pipelines/trigger` with `{ inputs: { provider: 'aws' } }` +1. Trigger a pipeline via `trigger:` syntax. + +```yaml +deploy-app: + trigger: + project: org/deployer + with: + provider: aws + deploy_environment: staging +``` + +To solve the problem of `Run Pipeline` UI form we could fully leverage the `spec:inputs` schema: + +```yaml +spec: + inputs: + concurrency: + default: 10 # displayed as default value in the input box + provider: + required: true # can enforce `required` in the form validation + deploy_environment: + options: # render a selectbox with options in order of how they are defined below + - staging # 1st option + - canary # 2nd option + - production # 3rd option + default: staging # selected by default in the UI. + # if `default:` is not specified, the user must explicitly select + # an option. + description: Deployment environment # optional: render as input label. +``` + ## Limits Any MVC that exposes a feature should be added with limitations from the beginning. -- GitLab From 921a9834550eaf1c24b1631fb7363d4489dcde62 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Mon, 5 Dec 2022 08:30:37 +0000 Subject: [PATCH 2/6] Feedback from review --- .../ci_pipeline_components/index.md | 54 ++++++++----------- 1 file changed, 23 insertions(+), 31 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index ded27ad93ad11a..14dfc4b4ab8120 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -177,12 +177,10 @@ A `gitlab-.yml` file: spec: description: My component description inputs: + website: environment: default: test - website: - required: true test_run: - required: true options: - unit - integration @@ -298,13 +296,11 @@ If the component takes any input parameters they must be specified according to ```yaml spec: inputs: + website: # by default all declared inputs are mandatory. environment: - default: test # apply default if not provided. - website: - required: true # fail if value not provided. + default: test # apply default if not provided. This makes the input optional. test_run: - required: true # a choice must be made from the list since there is no default value. - options: + options: # a choice must be made from the list since there is no default value. - unit - integration - system @@ -322,7 +318,7 @@ include: Input parameters are validated as soon as possible: -1. Read the file `.gitlab-template.yml` inside `org/my-component`. +1. Read the file `gitlab-template.yml` inside `org/my-component`. 1. Parse `spec:inputs` and validate the parameters against this schema. 1. If successfully validated, proceed with parsing `content:`. Return an error otherwise. 1. Interpolate input parameters inside the component's `content:`. @@ -333,27 +329,17 @@ spec: environment: options: [test, staging, production] content: - run-tests: + "run-tests-$[[ inputs.environment ]]": script: ./run-test - rules: - - if: $[[ inputs.environment ]] == 'test' - "scan-$[[ inputs.environment ]]-website": - script: ./scan-website ${{ inputs.environment }} + + scan-website: + script: ./scan-website $[[ inputs.environment ]] rules: - if: $[[ inputs.environment ]] == 'staging' - if: $[[ inputs.environment ]] == 'production' ``` -- `$[[ inputs.XXX ]]` syntax interpolates the inputs immediately after parsing the `content:`. -- `${{ inputs.XXX }}` syntax interpolates the inputs on the Runner, during runtime. - -> TODO: Do we really need this differentiation? -> ${{ ... }} cannot be used anywhere outside `script:`, `after_script:` and `before_script:`. -> We could instead benefit from interpolating values immediately and save `options[:script]` -> as expanded version. -> -> Runtime interpolation makes sense if the values change in the meantime, but that's not -> the case. +With `$[[ inputs.XXX ]]` inputs are interpolated immediately after parsing the `content:`. ### Why input parameters and not environment variables? @@ -365,8 +351,9 @@ Using environment variables for passing information to a component is like decla variables in programming languages. The more variables we declare the more we risk variable conflicts and increase variables scope. -Input parameters are like local variables that exist inside a specific scope and they don't -leak to the outside. +Input parameters are like variables passed to the component which exist inside a specific +scope and they don't leak to the outside. +Inputs are not inherited from upstream `include`s. They must be passed explicitly. This paradigm allows to build more robust and isolated components as well as declare and enforce contracts. @@ -388,10 +375,14 @@ include: file: .gitlab-ci.yml with: foo: bar + - remote: http://example.com/ci/config + with: + foo: bar + - template: Auto-DevOps.gitlab-ci.yml + with: + foo: bar ``` -> TODO: what about `template` and `remote`? - ### Input parameters for pipelines Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. @@ -400,7 +391,8 @@ Today we have different use cases where using explicit input parameters would be 1. `Run Pipeline` UI form. - **Problem today**: We are using top-level variables with `variables:*:description` to surface environment variables to the UI. - The problem with this is the mix of responsibilities as well as the jump in precedence that a variable gets (from job variable to pipeline variable). + The problem with this is the mix of responsibilities as well as the jump in [precedence](../../../ci/variables/index.md#cicd-variable-precedence) + that a variable gets (from a YAML variable to a pipeline variable). Building validation and features on top of this solution is challenging and complex. 1. Trigger a pipeline via API. For example `POST /projects/:id/pipelines/trigger` with `{ inputs: { provider: 'aws' } }` 1. Trigger a pipeline via `trigger:` syntax. @@ -421,8 +413,8 @@ spec: inputs: concurrency: default: 10 # displayed as default value in the input box - provider: - required: true # can enforce `required` in the form validation + provider: # can enforce `required` in the form validation + description: Deployment provider # optional: render as input label. deploy_environment: options: # render a selectbox with options in order of how they are defined below - staging # 1st option -- GitLab From d7acd894702ce503bc8cd93595fcce6690817e1f Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Mon, 5 Dec 2022 08:45:35 +0000 Subject: [PATCH 3/6] Clarify `with` and `inputs` relation --- .../blueprints/ci_pipeline_components/index.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 14dfc4b4ab8120..6fd68561bf0c01 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -383,6 +383,15 @@ include: foo: bar ``` +If a YAML includes content using `with:` but the including YAML doesn't specify `inputs:`, an error should be raised. + +|`with:`| `inputs:` | result | +| --- | --- | --- | +| specified | | raise error | +| specified | specified | validate inputs | +| | specified | use defaults | +| | | legacy `include:` without input passing | + ### Input parameters for pipelines Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. -- GitLab From 93924bc3d9d847d96f1d16b949639d2b48477c70 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Mon, 5 Dec 2022 11:23:34 +0000 Subject: [PATCH 4/6] Descope `spec:description` for now --- doc/architecture/blueprints/ci_pipeline_components/index.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 6fd68561bf0c01..10b58c54acfef8 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -175,7 +175,6 @@ A `gitlab-.yml` file: ```yaml spec: - description: My component description inputs: website: environment: -- GitLab From 94362c782d1e1b7ebfad757d0fbccda7442ab914 Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Tue, 6 Dec 2022 11:12:32 +0000 Subject: [PATCH 5/6] Support variables and inputs expansion --- doc/architecture/blueprints/ci_pipeline_components/index.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 10b58c54acfef8..06368edc7d2695 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -311,10 +311,14 @@ When using the component we pass the input parameters as follow: include: - component: org/my-component@1.0 with: - website: http://example.com + website: ${MY_WEBSITE} # variables expansion test_run: system + environment: $[[ inputs.environment ]] # interpolation of upstream inputs ``` +Variables expansion must be supported for `with:` syntax as well as interpolation of +possible [inputs provided upstream](#input-parameters-for-pipelines). + Input parameters are validated as soon as possible: 1. Read the file `gitlab-template.yml` inside `org/my-component`. -- GitLab From 2a23d745e82b449b69046298ff5c4b7fce8ed80e Mon Sep 17 00:00:00 2001 From: Fabio Pitino Date: Tue, 6 Dec 2022 12:07:03 +0000 Subject: [PATCH 6/6] Clarify use of spec:inputs in included file --- .../blueprints/ci_pipeline_components/index.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index 06368edc7d2695..ff0464fc01421d 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -305,7 +305,7 @@ spec: - system ``` -When using the component we pass the input parameters as follow: +When using the component we pass the input parameters as follows: ```yaml include: @@ -386,6 +386,16 @@ include: foo: bar ``` +Then the configuration being included must specify the inputs: + +```yaml +spec: + inputs: + foo: + +# rest of the configuration +``` + If a YAML includes content using `with:` but the including YAML doesn't specify `inputs:`, an error should be raised. |`with:`| `inputs:` | result | @@ -432,7 +442,7 @@ spec: - staging # 1st option - canary # 2nd option - production # 3rd option - default: staging # selected by default in the UI. + default: staging # selected by default in the UI. # if `default:` is not specified, the user must explicitly select # an option. description: Deployment environment # optional: render as input label. -- GitLab