From de4bddc3bcf94f92d7b06f02a75a145bf9c301e1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Elan=20Ruusam=C3=A4e?= Date: Mon, 16 Dec 2024 22:29:55 +0000 Subject: [PATCH] Doc: Fix append to two files in variables example --- doc/ci/variables/index.md | 530 ++++++++++++++++++++++++++++---------- 1 file changed, 392 insertions(+), 138 deletions(-) diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 34b3d35a4bc394..7b93d56318b1e6 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -1,11 +1,14 @@ --- stage: Verify -group: Pipeline Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments -type: reference +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments --- -# GitLab CI/CD variables **(FREE ALL)** +# GitLab CI/CD variables + +DETAILS: +**Tier:** Free, Premium, Ultimate +**Offering:** GitLab.com, Self-managed, GitLab Dedicated CI/CD variables are a type of environment variable. You can use them to: @@ -13,7 +16,7 @@ CI/CD variables are a type of environment variable. You can use them to: - Store values you want to re-use. - Avoid hard-coding values in your `.gitlab-ci.yml` file. -You can [override variable values manually for a specific pipeline](../jobs/index.md#specifying-variables-when-running-manual-jobs), +You can [override variable values](#cicd-variable-precedence) for a specific pipeline when you [run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually), [run a manual job](../jobs/index.md#specifying-variables-when-running-manual-jobs), or have them [prefilled in manual pipelines](../pipelines/index.md#prefill-variables-in-manual-pipelines). Variable names are limited by the [shell the runner uses](https://docs.gitlab.com/runner/shells/index.html) @@ -54,35 +57,44 @@ value with the [`variables`](../yaml/index.md#variables) keyword. Variables saved in the `.gitlab-ci.yml` file are visible to all users with access to the repository, and should store only non-sensitive project configuration. For example, the URL of a database saved in a `DATABASE_URL` variable. Sensitive variables containing values -like secrets or keys should be [stored in project settings](#define-a-cicd-variable-in-the-ui). +like secrets or keys should be [added in the UI](#define-a-cicd-variable-in-the-ui). -You can use `variables` in a job or at the top level of the `.gitlab-ci.yml` file. +You can define `variables` in a job or at the top level of the `.gitlab-ci.yml` file. If the variable is defined: -- At the top level, it's globally available and all jobs can use it. -- In a job, only that job can use it. +- In a job, only that job can use it. You can use the variable in the job's `script`, `before_script`, or `after_script` sections, + and also with some [job keywords](../yaml/index.md#job-keywords), but not [global keywords](../yaml/index.md#global-keywords). +- In a global (top-level) `variables` section, it acts as a default variable for all jobs. + Each global variable is made available to every job in the pipeline, except when the job already has a variable + defined with the same name. The variable defined in the job [takes precedence](../variables/index.md#cicd-variable-precedence), + so you cannot use the value of the global variable with the same name in the job. + + Like job variables, you cannot use global variables as values for other global keywords. For example: ```yaml variables: - GLOBAL_VAR: "A global variable" + ALL_JOBS_VAR: "A global variable" job1: variables: - JOB_VAR: "A job variable" + JOB1_VAR: "Job 1 variable" script: - - echo "Variables are '$GLOBAL_VAR' and '$JOB_VAR'" + - echo "Variables are '$ALL_JOBS_VAR' and '$JOB1_VAR'" job2: + variables: + ALL_JOBS_VAR: "Different value than global" + JOB2_VAR: "Job 2 variable" script: - - echo "Variables are '$GLOBAL_VAR' and '$JOB_VAR'" + - echo "Variables are '$ALL_JOBS_VAR', '$JOB2_VAR', and '$JOB1_VAR'" ``` In this example: -- `job1` outputs `Variables are 'A global variable' and 'A job variable'` -- `job2` outputs `Variables are 'A global variable' and ''` +- `job1` outputs: `Variables are 'A global variable' and 'Job 1 variable'` +- `job2` outputs: `Variables are 'Different value than global', 'Job 2 variable', and ''` Use the [`value` and `description`](../yaml/index.md#variablesdescription) keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) @@ -107,7 +119,7 @@ job1: Sensitive variables like tokens or passwords should be stored in the settings in the UI, not [in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). -Define CI/CD variables in the UI: +Add CI/CD variables in the UI: - For a project [in the project's settings](#for-a-project). - For all projects in a group [in the group's setting](#for-a-group). @@ -125,88 +137,106 @@ all variables become available to the pipeline. ### For a project -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can define a maximum of 200 CI/CD variables. -> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, projects can define a maximum of 8000 CI/CD variables. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, projects can have a maximum of 200 CI/CD variables. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, projects can have a maximum of 8000 CI/CD variables. You can add CI/CD variables to a project's settings. -Prerequisite: +Prerequisites: - You must be a project member with the Maintainer role. To add or update variables in the project settings: -1. Go to your project's **Settings > CI/CD** and expand the **Variables** section. +1. On the left sidebar, select **Search or go to** and find your project. +1. Select **Settings > CI/CD**. +1. Expand **Variables**. 1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - - **Environment scope**: Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). - - **Protect variable** Optional. If selected, the variable is only available - in pipelines that run on [protected branches](../../user/project/protected_branches.md) or [protected tags](../../user/project/protected_tags.md). - - **Mask variable** Optional. If selected, the variable's **Value** is masked - in job logs. The variable fails to save if the value does not meet the - [masking requirements](#mask-a-cicd-variable). - -After you create a variable, you can use it in the [`.gitlab-ci.yml` configuration](../yaml/index.md) + - **Environment scope**: Optional. **All (default)** (`*`), a specific [environment](../environments/index.md#types-of-environments), + or a wildcard [environment scope](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). + - **Protect variable** Optional. If selected, the variable is only available in pipelines + that run on [protected branches](../../user/project/repository/branches/protected.md) + or [protected tags](../../user/project/protected_tags.md). + - **Visibility**: Select **Visible** (default), [**Masked**](#mask-a-cicd-variable), + or [**Masked and hidden**](#hide-a-cicd-variable) (only available for new variables). + +After you create a variable, you can use it in the pipeline configuration or in [job scripts](#use-cicd-variables-in-job-scripts). ### For a group -> - Support for environment scopes [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2874) in GitLab Premium 13.11 -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, groups can define a maximum of 200 CI/CD variables. -> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, groups can define a maximum of 30000 CI/CD variables. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/362227) in GitLab 15.7, groups can have a maximum of 200 CI/CD variables. +> - [Updated](https://gitlab.com/gitlab-org/gitlab/-/issues/373289) in GitLab 15.9, groups can have a maximum of 30000 CI/CD variables. You can make a CI/CD variable available to all projects in a group. -Prerequisite: +Prerequisites: - You must be a group member with the Owner role. To add a group variable: -1. In the group, go to **Settings > CI/CD**. +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > CI/CD**. +1. Expand **Variables**. 1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - **Value**: No limitations. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - - **Environment scope** Optional. `All`, or specific [environments](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). **(PREMIUM ALL)** - - **Protect variable** Optional. If selected, the variable is only available - in pipelines that run on protected branches or tags. - - **Mask variable** Optional. If selected, the variable's **Value** is masked - in job logs. The variable fails to save if the value does not meet the - [masking requirements](#mask-a-cicd-variable). + - **Protect variable** Optional. If selected, the variable is only available in pipelines + that run on [protected branches](../../user/project/repository/branches/protected.md) + or [protected tags](../../user/project/protected_tags.md). + - **Visibility**: Select **Visible** (default), [**Masked**](#mask-a-cicd-variable), + or [**Masked and hidden**](#hide-a-cicd-variable) (only available for new variables). The group variables that are available in a project are listed in the project's **Settings > CI/CD > Variables** section. Variables from [subgroups](../../user/group/subgroups/index.md) are recursively inherited. -### For an instance **(FREE SELF)** +#### Environment scope + +DETAILS: +**Tier:** Premium, Ultimate + +To set a group CI/CD variable to only be available for certain environments: + +1. On the left sidebar, select **Search or go to** and find your group. +1. Select **Settings > CI/CD**. +1. Expand **Variables**. +1. To the right of the variable, select **Edit** (**{pencil}**). +1. For **Environment scope**, select **All (default)** (`*`), a specific [environment](../environments/index.md#types-of-environments), + or a wildcard [environment scope](../environments/index.md#limit-the-environment-scope-of-a-cicd-variable). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14108) in GitLab 13.0. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/299879) in GitLab 13.11. +### For an instance + +DETAILS: +**Tier:** Free, Premium, Ultimate +**Offering:** Self-managed, GitLab Dedicated You can make a CI/CD variable available to all projects and groups in a GitLab instance. -Prerequisite: +Prerequisites: - You must have administrator access to the instance. To add an instance variable: -1. On the left sidebar, select **Search or go to**. -1. Select **Admin Area**. -1. Select **Settings > CI/CD** and expand the **Variables** section. +1. On the left sidebar, at the bottom, select **Admin**. +1. Select **Settings > CI/CD**. +1. Expand **Variables**. 1. Select **Add variable** and fill in the details: - **Key**: Must be one line, with no spaces, using only letters, numbers, or `_`. - - **Value**: In [GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/220028), - the value is limited to 10,000 characters, but also bounded by any limits in the - runner's operating system. In GitLab 13.0 to 13.2, the value is limited to 700 characters. + - **Value**: The value is limited to 10,000 characters, but also bounded by any limits in the + runner's operating system. - **Type**: `Variable` (default) or [`File`](#use-file-type-cicd-variables). - - **Protect variable** Optional. If selected, the variable is only available - in pipelines that run on protected branches or tags. - - **Mask variable** Optional. If selected, the variable's **Value** is not shown - in job logs. The variable is not saved if the value does not meet the [masking requirements](#mask-a-cicd-variable). + - **Protect variable** Optional. If selected, the variable is only available in pipelines + that run on [protected branches](../../user/project/repository/branches/protected.md) + or [protected tags](../../user/project/protected_tags.md). + - **Visibility**: Select **Visible** (default), [**Masked**](#mask-a-cicd-variable), + or [**Masked and hidden**](#hide-a-cicd-variable) (only available for new variables). ## CI/CD variable security @@ -234,11 +264,18 @@ malicious-job: ``` To help reduce the risk of accidentally leaking secrets through scripts like in `accidental-leak-job`, -all variables containing sensitive information should be [masked in job logs](#mask-a-cicd-variable). +all variables containing sensitive information should always be [masked in job logs](#mask-a-cicd-variable). You can also [limit a variable to protected branches and tags only](#protect-a-cicd-variable). -Alternatively, use the GitLab [integration with HashiCorp Vault](../secrets/index.md) -to store and retrieve secrets. +Alternatively, use one of the native GitLab integrations to connect with third party +secrets manager providers to store and retrieve secrets: + +- [HashiCorp Vault](../secrets/index.md) +- [Azure Key Vault](../secrets/azure_key_vault.md) +- [Google Secret Manager](../secrets/gcp_secret_manager.md) + +You can also use [OpenID Connect (OIDC) authentication](../secrets/id_token_authentication.md) +for secrets managers which do not have a native integration. Malicious scripts like in `malicious-job` must be caught during the review process. Reviewers should never trigger a pipeline when they find code like this, because @@ -246,42 +283,50 @@ malicious code can compromise both masked and protected variables. Variable values are encrypted using [`aes-256-cbc`](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) and stored in the database. This data can only be read and decrypted with a -valid [secrets file](../../administration/backup_restore/backup_gitlab.md#when-the-secrets-file-is-lost). +valid [secrets file](../../administration/backup_restore/troubleshooting_backup_gitlab.md#when-the-secrets-file-is-lost). ### Mask a CI/CD variable -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330650) in GitLab 13.12, the `~` character can be used in masked variables. - WARNING: Masking a CI/CD variable is not a guaranteed way to prevent malicious users from -accessing variable values. The masking feature is "best-effort" and there to -help when a variable is accidentally revealed. To make variables more secure, +accessing variable values. To ensure security of sensitive information, consider using [external secrets](../secrets/index.md) and [file type variables](#use-file-type-cicd-variables) to prevent commands such as `env`/`printenv` from printing secret variables. You can mask a project, group, or instance CI/CD variable so the value of the variable -does not display in job logs. +does not display in job logs. When a masked CI/CD variable would be displayed in a job log, +the value is replaced with `[masked]` to prevent the value from being exposed. -Prerequisite: +Prerequisites: -- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). +- You must have the same role or access level as required to [add a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). To mask a variable: -1. In the project, group, or Admin Area, go to **Settings > CI/CD**. -1. Expand the **Variables** section. +1. For the group, project, or in the **Admin** area, select **Settings > CI/CD**. +1. Expand **Variables**. 1. Next to the variable you want to protect, select **Edit**. -1. Select the **Mask variable** checkbox. +1. Under **Visibility**, select **Mask variable**. 1. Select **Update variable**. The method used to mask variables [limits what can be included in a masked variable](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784#note_106756757). The value of the variable must: -- Be a single line. -- Be 8 characters or longer, consisting only of: - - Characters from the Base64 alphabet (RFC4648). - - The `@`, `:`, `.`, or `~` characters. +- Be a single line with no spaces. +- Be 8 characters or longer. - Not match the name of an existing predefined or custom CI/CD variable. +- Not include non-alphanumeric characters other than `@`, `_`, `-`, `:`, or `+`. + +Additionally, if [variable expansion](#prevent-cicd-variable-expansion) is enabled, +the value can contain only: + +- Characters from the Base64 alphabet (RFC4648). +- The `@`, `:`, `.`, or `~` characters. + +Masking a variable automatically masks the value anywhere in a job log. If another +variable has the same value, that value is also masked, including when a variable +references a masked variable. The string `[MASKED]` is shown instead of the value, +possibly with some trailing `x` characters. Different versions of [GitLab Runner](../runners/index.md) have different masking limitations: @@ -291,25 +336,41 @@ Different versions of [GitLab Runner](../runners/index.md) have different maskin | v14.2.0 to v15.3.0 | The tail of a large secret (greater than 4 KiB) could potentially be [revealed](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28128). No sensitive URL parameter masking. | | v15.7.0 and later | Secrets could be revealed when `CI_DEBUG_SERVICES` is enabled. For details, read about [service container logging](../services/index.md#capturing-service-container-logs). | +### Hide a CI/CD variable + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29674) in GitLab 17.4 [with a flag](../../administration/feature_flags.md) named `ci_hidden_variables`. Enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/165843) in GitLab 17.6. Feature flag `ci_hidden_variables` removed. + +In addition to masking, you can also prevent the value of CI/CD variables from being revealed +in the **CI/CD** settings page. Hiding a variable is only possible when creating a new variable, +you cannot update an existing variable to be hidden. + +Prerequisites: + +- You must have the same role or access level as required to [add a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). +- The variable value must match the [requirements for masked variables](#mask-a-cicd-variable). + +To hide a variable, select **Masked and hidden** in the **Visibility** section when +you [add a new CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). +After you save the variable, the variable can be used in CI/CD pipelines, but cannot +be revealed in the UI again. + ### Protect a CI/CD variable You can configure a project, group, or instance CI/CD variable to be available -only to pipelines that run on [protected branches](../../user/project/protected_branches.md) +only to pipelines that run on [protected branches](../../user/project/repository/branches/protected.md) or [protected tags](../../user/project/protected_tags.md). -[Merged results pipelines](../pipelines/merged_results_pipelines.md), which run on a -temporary merge commit, not a branch or tag, do not have access to these variables. -[Merge request pipelines](../pipelines/merge_request_pipelines.md), which do not use -a temporary merge commit, can access these variables if the branch is a protected branch. +[Merged results pipelines](../pipelines/merged_results_pipelines.md) and [merge request pipelines](../pipelines/merge_request_pipelines.md) do not have access to these variables. -Prerequisite: +Prerequisites: -- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). +- You must have the same role or access level as required to [add a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). To set a variable as protected: -1. Go to **Settings > CI/CD** in the project, group or instance Admin Area. -1. Expand the **Variables** section. +1. For the project or group, go to **Settings > CI/CD**. +1. Expand **Variables**. 1. Next to the variable you want to protect, select **Edit**. 1. Select the **Protect variable** checkbox. 1. Select **Update variable**. @@ -356,7 +417,7 @@ kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KU WARNING: Be careful when assigning the value of a file variable to another variable in GitLab 15.6 or older. The other variable takes the content of the file as its value, **not** the path to the file. -In GitLab 15.7 and newer, this behavior [was fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) and the other variable now takes the path to the file as the value. +In GitLab 15.7 and later, this behavior [was fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/29407) and the other variable now takes the path to the file as the value. #### Use a `.gitlab-ci.yml` variable as a file type variable @@ -371,7 +432,7 @@ For example: ```yaml variables: - SITE_URL: "https://example.gitlab.com" + SITE_URL: "https://gitlab.example.com" job: script: @@ -443,24 +504,24 @@ job_name: [Service containers](../docker/using_docker_images.md) can use CI/CD variables, but by default can only access [variables saved in the `.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). +Variables [added in the GitLab UI](#define-a-cicd-variable-in-the-ui) are not available to +service containers, because service containers are not trusted by default. -Variables [set in the GitLab UI](#define-a-cicd-variable-in-the-ui) by default are not available to -service containers. To make a UI-defined variable available in a service container, -re-assign it in your `.gitlab-ci.yml`: +To make a UI-defined variable available in a service container, you can re-assign +it to another variable in your `.gitlab-ci.yml`: ```yaml variables: SA_PASSWORD_YAML_FILE: $SA_PASSWORD_UI ``` -### Pass an environment variable to another job +The re-assigned variable cannot have the same name as the original variable. Otherwise it does not get expanded. -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22638) in GitLab 13.0. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217834) in GitLab 13.1. +### Pass an environment variable to another job -You can create a new environment variables in a job, and pass it to another job -in a later stage. These variables cannot be used as CI/CD variables to configure a pipeline, -but they can be used in job scripts. +You can create a new environment variable in a job, and pass it to another job +in a later stage. These variables cannot be used as CI/CD variables to configure a pipeline +(for example with the [`rules` keyword](../yaml/index.md#rules)), but they can be used in job scripts. To pass a job-created environment variable to other jobs: @@ -493,7 +554,7 @@ test-job: Variables from `dotenv` reports [take precedence](#cicd-variable-precedence) over certain types of new variable definitions such as job defined variables. -You can also [pass `dotenv` variables to downstream pipelines](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job) +You can also [pass `dotenv` variables to downstream pipelines](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job). #### Control which jobs receive `dotenv` variables @@ -528,7 +589,7 @@ test-job1: script: - echo "$BUILD_VERSION" # Output is: 'v1.0.0' dependencies: - - build + - build-job1 test-job2: stage: test @@ -548,16 +609,16 @@ test-job4: script: - echo "$BUILD_VERSION" # Output is: 'v1.0.0' needs: - job: build-job1 - artifacts: true + - job: build-job1 + artifacts: true test-job5: stage: deploy script: - echo "$BUILD_VERSION" # Output is '' needs: - job: build-job1 - artifacts: false + - job: build-job1 + artifacts: false test-job6: stage: deploy @@ -567,6 +628,43 @@ test-job6: - build-job2 ``` +### Pass an environment variable from the `script` section to another section in the same job + +Use `$GITLAB_ENV` to pass environment variables defined in the `script` section to another section. + +For example: + +```yaml +build-job: + stage: build + script: + - echo "ARCH=$(arch)" >> $GITLAB_ENV + - touch some-file-$(arch) + artifacts: + paths: + - some-file-$ARCH +``` + +To also reference the variable in other stages, write the variable to both the `$GITLAB_ENV` and `.env` files: + +```yaml +build-job: + stage: build + script: + - echo "ARCH=$(arch)" | tee -a $GITLAB_ENV >> build.env + - touch some-file-$(arch) + artifacts: + paths: + - some-file-$ARCH + reports: + dotenv: build.env + +release-job: + stage: release + script: + - curl --upload-file some-file-$ARCH "https://example.com/some-file-$ARCH" +``` + ### Store multiple values in one variable You cannot create a CI/CD variable that is an array of values, but you @@ -587,6 +685,24 @@ job1: done ``` +### As part of a string + +You can use variables as part of a string. You can surround the variables with curly brackets (`{}`) +to help distinguish the variable name from the surrounding text. Without curly brackets, +the adjacent text is interpreted as part of the variable name. For example: + +```yaml +job: + variables: + FLAGS: '-al' + DIR: 'path/to/directory' + LS_CMD: 'ls "$FLAGS"' + CD_CMD: 'cd "${DIR}_files"' + script: + - 'eval "$LS_CMD"' # Executes 'ls -al' + - 'eval "$CD_CMD"' # Executes 'cd path/to/directory_files' +``` + ## Use CI/CD variables in other variables You can use variables inside other variables: @@ -616,37 +732,43 @@ job: ### Prevent CI/CD variable expansion -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 15.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217309) in GitLab 15.7. Expanded variables treat values with the `$` character as a reference to another variable. CI/CD variables are expanded by default. To treat variables with a `$` character as raw strings, disable variable expansion for the variable -Prerequisite: +Prerequisites: -- You must have the same role or access level as required to [define a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). +- You must have the same role or access level as required to [add a CI/CD variable in the UI](#define-a-cicd-variable-in-the-ui). To disable variable expansion for the variable: -1. In the project or group, go to **Settings > CI/CD**. -1. Expand the **Variables** section. +1. For the project or group, go to **Settings > CI/CD**. +1. Expand **Variables**. 1. Next to the variable you want to do not want expanded, select **Edit**. 1. Clear the **Expand variable** checkbox. 1. Select **Update variable**. ## CI/CD variable precedence +> - Scan Execution Policies variable precedence was [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/424028) in GitLab 16.7 [with a flag](../../administration/feature_flags.md) named `security_policies_variables_precedence`. Enabled by default. [Feature flag removed in GitLab 16.8](https://gitlab.com/gitlab-org/gitlab/-/issues/435727). + You can use CI/CD variables with the same name in different places, but the values can overwrite each other. The type of variable and where they are defined determines which variables take precedence. The order of precedence for variables is (from highest to lowest): -1. These variables all have the same (highest) precedence: +1. [Pipeline execution policy variables](../../user/application_security/policies/pipeline_execution_policies.md#cicd-variables). +1. [Scan execution policy variables](../../user/application_security/policies/scan_execution_policies.md). +1. [Pipeline variables](#use-pipeline-variables). These variables all have the same precedence: + - [Variables passed to downstream pipelines](../pipelines/downstream_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline). - [Trigger variables](../triggers/index.md#pass-cicd-variables-in-the-api-call). - [Scheduled pipeline variables](../pipelines/schedules.md#add-a-pipeline-schedule). - [Manual pipeline run variables](../pipelines/index.md#run-a-pipeline-manually). - Variables added when [creating a pipeline with the API](../../api/pipelines.md#create-a-new-pipeline). + - [Manual job variables](../jobs/index.md#specifying-variables-when-running-manual-jobs). 1. Project [variables](#for-a-project). 1. Group [variables](#for-a-group). If the same variable name exists in a group and its subgroups, the job uses the value from the closest subgroup. For example, if @@ -672,37 +794,131 @@ job1: - echo "The variable is '$API_TOKEN'" ``` -In this example, `job1` outputs `The variable is 'secure'` because variables defined in jobs -have higher precedence than variables defined globally. +In this example, `job1` outputs `The variable is 'secure'` because variables defined in jobs in the `.gitlab-ci.yml` file +have higher precedence than variables defined globally in the `.gitlab-ci.yml` file. + +## Use pipeline variables -### Override a defined CI/CD variable +Pipeline variables are variables that are specified when running a new pipeline. -You can override the value of a variable when you: +Prerequisites: + +- You must have the Developer role in the project. + +You can specify a pipeline variable when you: - [Run a pipeline manually](../pipelines/index.md#run-a-pipeline-manually) in the UI. - Create a pipeline by using [the `pipelines` API endpoint](../../api/pipelines.md#create-a-new-pipeline). -- Use [push options](../../user/project/push_options.md#push-options-for-gitlab-cicd). -- Trigger a pipeline by using [the `triggers` API endpoint](../triggers/index.md#pass-cicd-variables-in-the-api-call). -- Pass variables to a downstream pipeline [by using the `variable` keyword](../pipelines/downstream_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline) - or [by using `dotenv` variables](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job). +- Create a pipeline by using [the `triggers` API endpoint](../triggers/index.md#pass-cicd-variables-in-the-api-call). +- Use [push options](../../topics/git/commit.md#push-options-for-gitlab-cicd). +- Pass variables to a downstream pipeline by using either the [`variables` keyword](../pipelines/downstream_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline), + [`trigger:forward` keyword](../yaml/index.md#triggerforward) or [`dotenv` variables](../pipelines/downstream_pipelines.md#pass-dotenv-variables-created-in-a-job). +- Specify variables when creating a [pipeline schedule](../pipelines/schedules.md#add-a-pipeline-schedule). +- Specify variables when [running a manual job](../pipelines/index.md#run-a-pipeline-manually). + +These variables have [higher precedence](#cicd-variable-precedence) and can override +other defined variables, including [predefined variables](predefined_variables.md). -You should avoid overriding [predefined variables](predefined_variables.md), as it -can cause the pipeline to behave unexpectedly. +WARNING: +You should avoid overriding predefined variables in most cases, as it can cause the pipeline to behave unexpectedly. -### Restrict who can override variables +### Restrict pipeline variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295234) in GitLab 13.8. +You can limit who can run pipelines with pipeline variables to specific user roles. +To limit the use of pipeline variables to only the Maintainer role and higher: -You can limit the ability to override variables to only users with the Maintainer role. -When other users try to run a pipeline with overridden variables, they receive the -`Insufficient permissions to set pipeline variables` error message. +- Use [the projects API](../../api/projects.md#edit-a-project) to enable the `restrict_user_defined_variables` setting. + The setting is `disabled` by default. -Enable this feature by using [the projects API](../../api/projects.md#edit-project) -to enable the `restrict_user_defined_variables` setting. The setting is `disabled` by default. +When users with the Developer role or lower try to [use pipeline variables](#use-pipeline-variables), +they receive the `Insufficient permissions to set pipeline variables` error message. If you [store your CI/CD configurations in a different repository](../../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file), use this setting for control over the environment the pipeline runs in. +#### Set a minimum role for pipeline variables + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/440338) in GitLab 17.1. +> - For GitLab.com, setting defaults [updated for all new projects in new namespaces](https://gitlab.com/gitlab-org/gitlab/-/issues/502382) to `enabled` for `restrict_user_defined_variables` and `no_one_allowed` for `ci_pipeline_variables_minimum_override_role` in GitLab 17.7. + +Prerequisites: + +- You must have the Maintainer role in the project. If the minimum role was previously set to `owner` + or `no_one_allowed`, then you must have the Owner role in the project. + +To change the setting, use [the projects API](../../api/projects.md#edit-a-project) +to set `ci_pipeline_variables_minimum_override_role` to one of: + +- `no_one_allowed`: No pipelines can run with pipeline variables. + Default for new projects in new namespaces on GitLab.com. +- `owner`: Only users with the Owner role can run pipelines with pipeline variables. + You must have the Owner role for the project to change the setting to this value. +- `maintainer`: Only users with at least the Maintainer role can run pipelines with pipeline variables. + Default when not specified on self-managed and Dedicated. +- `developer`: Only users with at least the Developer role can run pipelines with pipeline variables. + +## Exporting variables + +Scripts executed in separate shell contexts do not share exports, aliases, +local function definitions, or any other local shell updates. + +This means that if a job fails, variables created by user-defined scripts are not +exported. + +When runners execute jobs defined in `.gitlab-ci.yml`: + +- Scripts specified in `before_script` and the main script are executed together in + a single shell context, and are concatenated. +- Scripts specified in `after_script` run in a shell context completely separate to + the `before_script` and the specified scripts. + +Regardless of the shell the scripts are executed in, the runner output includes: + +- Predefined variables. +- Variables defined in: + - Instance, group, or project CI/CD settings. + - The `.gitlab-ci.yml` file in the `variables:` section. + - The `.gitlab-ci.yml` file in the `secrets:` section. + - The `config.toml`. + +The runner cannot handle manual exports, shell aliases, and functions executed in the body of the script, like `export MY_VARIABLE=1`. + +For example, in the following `.gitlab-ci.yml` file, the following scripts are defined: + +```yaml +job: + variables: + JOB_DEFINED_VARIABLE: "job variable" + before_script: + - echo "This is the 'before_script' script" + - export MY_VARIABLE="variable" + script: + - echo "This is the 'script' script" + - echo "JOB_DEFINED_VARIABLE's value is ${JOB_DEFINED_VARIABLE}" + - echo "CI_COMMIT_SHA's value is ${CI_COMMIT_SHA}" + - echo "MY_VARIABLE's value is ${MY_VARIABLE}" + after_script: + - echo "JOB_DEFINED_VARIABLE's value is ${JOB_DEFINED_VARIABLE}" + - echo "CI_COMMIT_SHA's value is ${CI_COMMIT_SHA}" + - echo "MY_VARIABLE's value is ${MY_VARIABLE}" +``` + +When the runner executes the job: + +1. `before_script` is executed: + 1. Prints to the output. + 1. Defines the variable for `MY_VARIABLE`. +1. `script` is executed: + 1. Prints to the output. + 1. Prints the value of `JOB_DEFINED_VARIABLE`. + 1. Prints the value of `CI_COMMIT_SHA`. + 1. Prints the value of `MY_VARIABLE`. +1. `after_script` is executed in a new, separate shell context: + 1. Prints to the output. + 1. Prints the value of `JOB_DEFINED_VARIABLE`. + 1. Prints the value of `CI_COMMIT_SHA`. + 1. Prints an empty value of `MY_VARIABLE`. The variable value cannot be detected because `after_script` is in a separate shell context to `before_script`. + ## Related topics - You can configure [Auto DevOps](../../topics/autodevops/index.md) to pass CI/CD variables @@ -730,7 +946,7 @@ use this setting for control over the environment the pipeline runs in. You can list all variables available to a script with the `export` command in Bash or `dir env:` in PowerShell. This exposes the values of **all** available variables, which can be a [security risk](#cicd-variable-security). -[Masked variables](#mask-a-cicd-variable) display as `[masked]`. +[Masked variables](#mask-a-cicd-variable) display as `[MASKED]`. For example, with Bash: @@ -747,13 +963,13 @@ export CI_JOB_ID="50" export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" export CI_COMMIT_SHORT_SHA="1ecfd275" export CI_COMMIT_REF_NAME="main" -export CI_REPOSITORY_URL="https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab.git" +export CI_REPOSITORY_URL="https://gitlab-ci-token:[MASKED]@example.com/gitlab-org/gitlab.git" export CI_COMMIT_TAG="1.0.0" export CI_JOB_NAME="spec:other" export CI_JOB_STAGE="test" export CI_JOB_MANUAL="true" export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="[masked]" +export CI_JOB_TOKEN="[MASKED]" export CI_PIPELINE_ID="1000" export CI_PIPELINE_IID="10" export CI_PAGES_DOMAIN="gitlab.io" @@ -769,7 +985,7 @@ export CI_PROJECT_TITLE="GitLab" WARNING: Debug logging can be a serious security risk. The output contains the content of -all variables and other secrets available to the job. The output is uploaded to the +all variables available to the job. The output is uploaded to the GitLab server and visible in job logs. You can use debug logging to help troubleshoot problems with pipeline configuration @@ -857,8 +1073,8 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ++ CI_SERVER_PROTOCOL=https ++ export CI_SERVER_NAME=GitLab ++ CI_SERVER_NAME=GitLab -++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal -++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,visual_review_app,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ export GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal +++ GITLAB_FEATURES=audit_events,burndown_charts,code_owners,contribution_analytics,description_diffs,elastic_search,group_bulk_edit,group_burndown_charts,group_webhooks,issuable_default_templates,issue_weights,jenkins_integration,ldap_group_sync,member_lock,merge_request_approvers,multiple_issue_assignees,multiple_ldap_servers,multiple_merge_request_assignees,protected_refs_for_users,push_rules,related_issues,repository_mirrors,repository_size_limit,scoped_issue_board,usage_quotas,wip_limits,adjourned_deletion_for_projects_and_groups,admin_audit_log,auditor_user,batch_comments,blocking_merge_requests,board_assignee_lists,board_milestone_lists,ci_cd_projects,cluster_deployments,code_analytics,code_owner_approval_required,commit_committer_check,cross_project_pipelines,custom_file_templates,custom_file_templates_for_namespace,custom_project_templates,custom_prometheus_metrics,cycle_analytics_for_groups,db_load_balancing,default_project_deletion_protection,dependency_proxy,deploy_board,design_management,email_additional_text,extended_audit_events,external_authorization_service_api_management,feature_flags,file_locks,geo,github_integration,group_allowed_email_domains,group_project_templates,group_saml,issues_analytics,jira_dev_panel_integration,ldap_group_sync_filter,merge_pipelines,merge_request_performance_metrics,merge_trains,metrics_reports,multiple_approval_rules,multiple_group_issue_boards,object_storage,operations_dashboard,packages,productivity_analytics,project_aliases,protected_environments,reject_unsigned_commits,required_ci_templates,scoped_labels,service_desk,smartcard_auth,group_timelogs,type_of_work_analytics,unprotection_restrictions,ci_project_subscriptions,cluster_health,container_scanning,dast,dependency_scanning,epics,group_ip_restriction,incident_management,insights,license_management,personal_access_token_expiration_policy,pod_logs,prometheus_alerts,report_approver_rules,sast,security_dashboard,tracing,web_ide_terminal ++ export CI_PROJECT_ID=17893 ++ CI_PROJECT_ID=17893 ++ export CI_PROJECT_NAME=ci-debug-trace @@ -866,14 +1082,9 @@ if [[ -d "/builds/gitlab-examples/ci-debug-trace/.git" ]]; then ... ``` -#### Restrict access to debug logging - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213159) in GitLab 13.7. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/292661) in GitLab 13.8. +#### Access to debug logging -You can restrict access to debug logging. When restricted, only users with -at least the Developer role -can view job logs when debug logging is enabled with a variable in: +Access to debug logging is restricted to [users with at least the Developer role](../../user/permissions.md#cicd). Users with a lower role cannot see the logs when debug logging is enabled with a variable in: - The [`.gitlab-ci.yml` file](#define-a-cicd-variable-in-the-gitlab-ciyml-file). - The CI/CD variables set in the GitLab UI. @@ -883,10 +1094,6 @@ If you add `CI_DEBUG_TRACE` as a local variable to runners, debug logs generate to all users with access to job logs. The permission levels are not checked by the runner, so you should only use the variable in GitLab itself. -## Known issues and workarounds - -These are some know issues with CI/CD variables, and where applicable, known workarounds. - ### "argument list too long" This issue occurs when the combined length of all CI/CD variables defined for a job exceeds the limit imposed by the @@ -900,4 +1107,51 @@ As a workaround you can either: - Use [File-type](#use-file-type-cicd-variables) CI/CD variables for large environment variables where possible. - If a single large variable is larger than `ARG_MAX`, try using [Secure Files](../secure_files/index.md), or -bring the file to the job through some other mechanism. + bring the file to the job through some other mechanism. + +### Global variable doesn't expand in job variable of the same name + +You cannot use a global variable's value in a job variable of the same name. A global variable +is only made available to a job when the job does not have a variable defined with the same name. +If the job has a variable with the same name, the job's variable takes precedence +and the global variable is not available in the job. + +For example, these two samples are equivalent: + +- In this sample, `$MY_VAR` has no value because it's not defined anywhere: + + ```yaml + Job-with-variable: + variables: + MY_VAR: $MY_VAR + script: echo "Value is '$MY_VAR'" + ``` + +- In this sample, `$MY_VAR` has no value because the global variable with the same name + is not available in the job: + + ```yaml + variables: + MY_VAR: "Global value" + + Job-with-same-name-variable: + variables: + MY_VAR: $MY_VAR + script: echo "Value is '$MY_VAR'" + ``` + +In both cases, the echo command outputs `Value is '$MY_VAR'`. + +In general, you should use the global variable directly in a job rather than reassigning its value to a new variable. +If you need to do this, use variables with different names instead. For example: + +```yaml +variables: + MY_VAR1: "Global value1" + MY_VAR2: "Global value2" + +overwrite-same-name: + variables: + MY_VAR2_FROM_GLOBALS: $MY_VAR2 + script: echo "Values are '$MY_VAR1' and '$MY_VAR2_FROM_GLOBALS'" +``` -- GitLab