[go: up one dir, main page]
Skip to content

Backend: Allow `needs` to be used with `rules`

Release notes

Use needs in rules to update a job's needs for specific conditions. When a condition matches a rule, the job's needs configuration is completely replaced with the needs in the rule

Problem to solve

Original description * Introduction of Rules and Needs features provide some incredibly powerful tools. * Implementing Needs becomes a task that requires duplicating a massive number of downstream jobs if they can't be dynamically set.

A single downstream, with 2 optional presteps, quickly becomes 4 individual jobs that have to be mutually exclusive.

deploy release production:
  extends:
    - .deploy
    - .release
    - .production

deploy release production w/ prestep 1:
  extends:
    - .deploy
    - .release
    - .production
    - .prestep1

deploy release production w/ prestep 2:
  extends:
    - .deploy
    - .release
    - .production
    - .prestep2


deploy release production w/ prestep all:
  extends:
    - .deploy
    - .release
    - .production
    - .prestepall

Allowing needs to be set with rules conditions can easily cut this down to one job that is much more predictable

deploy release production:
  extends:
    - .deploy
    - .release
    - .production
  rules:
    # rules allows for a list of individual rule objects to be evaluated in order,
    # until one matches and dynamically provides attributes to the job.
    - if: '$DEPLOY_PRESTEP1 && $DEPLOY_PRESTEP2'
      needs: ["prestep1:release:production","prestep2:release:production"]
      when: on_success
    - if: '$DEPLOY_PRESTEP1'
      needs: ["prestep1:release:production"]
      when: on_success
    - if: '$DEPLOY_PRESTEP2'
      needs: ["prestep2:release:production"]
      when: on_success
  when: manual

Goal: We will allow rules to take needs which will be the exact set of keywords that we have under job's needs currently, see doc.

Intended users

  • DevOps Engineer - flexible yet predictable templating and abstraction

Further details

  • Reduce complexity and increase readability by reducing number of jobs.

What does success look like, and how can we measure that?

Definition of done: We can apply needs to rules with all the keywords that currently are available under job's needs.

Original proposal ### Proposal

Support the following syntax

test:
  rules:
    - if: '$SOMETING1'
      needs: ["job1"]
    - if: '$SOMETING2'
      needs: ["job2"]

When '$SOMETHING1' is true, the test job needs will change to ["job1"]
When '$SOMETHING2' is true, the test job needs will change to ["job2"]
When neither 'SOMETHING1' or 'SOMETHING2' evaluates to true, the job wont have any needs defined

test:
  needs: ['lint']
  rules:
    - if: '$SOMETHING1'
      needs: ["job1"]
    - if: '$SOMETHING2'
      needs: ["job2"]

When '$SOMETHING1' is true, the test job needs will change to ["job1"]
When '$SOMETHING2' is true, the test job needs will change to ["job2"]
When neither 'SOMETHING1' or 'SOMETHING2' evaluates to true, the needs will remain on ['lint']

if '$SOMETHING1' is true:

test:
  needs: ['lint']
  rules:
    - if: '$SOMETHING1'
      needs: 
        - job: "job1"
          optional: true
    - if: '$SOMETHING2'
      needs: ["job2"]

it evaluates to:

test:
  needs: 
    - job: ["job1"]
      optional: true

Rules are evaluated when the pipeline is created, and evaluated in order until the first match

Technical approach: The job's needs logic will not be copied, but reused under rules needs so in the future any changes done to needs on the job's side, will be reflected under the rules needs and vice versa

Permissions and Security

N/A

MR addressing this issue. FF issue.

Updated proposal

Currently we have needs on jobs eg

build:
 stage: build
 script: echo
 needs: [test]

In this issue, we are introducing needs on rules eg

lint_job:
  stage: lint
  script: 'echo lint_job'
  needs: [test]
  rules:
    - if: $var == null
      needs: [test_job]

or

lint_job:
  stage: lint
  script: 'echo lint_job'
  needs: [test]
  rules:
    - if: $var == null
      needs: 
        - job: test_job
          artifacts: true
          optional: false

If in the above lint_job example, if the rule evaluates to true, the job's need gets overridden to test_job. (In the later example, test_job attributes are also set: artifacts is set to true and option is set to false, if not specified, the default value is set)

Once specific condition in the rule is met, the needs from the rule override the needs on the job.

We will also allow the following keywords in the needs: artifacts, optional.

In this issue, we will not support the following options:

  1. https://docs.gitlab.com/ee/ci/yaml/#needsproject -> adding dependencies to download artifacts.
  2. https://docs.gitlab.com/ee/ci/yaml/#needspipeline -> provides mirroring pipeline status from an upstream pipeline to a bridge
  3. https://docs.gitlab.com/ee/ci/yaml/#needspipelinejob -> also downloading artifacts from a job in its parent pipeline or another child pipeline

Documentation

https://docs.gitlab.com/ee/ci/yaml/#permitted-attributes

Testing

  • Create a job who's needs can be controlled using variable flags.
  • Toggle different variable flags.

Alternative testing: Create yml file with a pipeline and set rules to evaluate to true, push the code. Once the code is pushed, the yml will automatically run and the jobs needs to be evaluated with considerations on job needs.

What does success look like, and how can we measure that?

Definition of done: We can apply needs to rules with artifacts, optional keywords.

Links / references

https://gitlab.my.salesforce.com/0016100001beDS0

Edited by Dov Hershkovitch