[go: up one dir, main page]
Skip to content

Improving canceling redundant pipelines with `interruptible`

Problem

Feature: Auto-cancel redundant pipelines.

The current behavior

The default of interruptible is false for jobs. This suggests that all jobs are uninterruptible unless the default section or a specific job in .gitlab-ci.yml is set to interruptible: true.

There is a scenario where, currently, this is not true, a pipeline can still be auto-canceled when no job is started.

As an example;

# https://gitlab.com/furkanayhan/test-project/-/blob/7ddf44269d7c310b030338d364124d81b6c3d1c9/.gitlab-ci.yml

test1:
  tags: [jahdkjashd] # invalid tag to keep this pending
  script: sleep 60

test2:
  tags: [jahdkjashd] # invalid tag to keep this pending
  script: sleep 60

When I send another commit to the same branch, these jobs will be canceled even if they don't have interruptible: true;

https://gitlab.com/furkanayhan/test-project/-/pipelines/878843139

Screenshot_2023-05-25_at_13.45.24

This is a documented behavior;

To completely cancel a running pipeline, all jobs must have interruptible: true, or interruptible: false jobs must not have started.

Its effect on downstream pipelines

Even though the behavior above can be expected behavior because of this sentence "Jobs that have not started yet (pending) are considered interruptible and safe to be cancelled.", it may raise unwanted behaviors for downstream pipelines for some users.

Steps and scenario:

  1. Let's say we only have two Runners. (In your runner's config.toml, set concurrent = 2 in the global section)
  2. Enable Auto-cancel redundant pipelines on your project
  3. Add a basic child pipeline in a file (.child.yml).
# .child.yml

build:
  stage: build
  script:
    - echo "Do your build here"
  1. Add the following .gitlab-ci.yml file to the project:
normal job:
  script: sleep 15

trigger a:
  trigger:
    include: ".child.yml"
    strategy: depend

trigger b:
  trigger:
    include: ".child.yml"
    strategy: depend

trigger c:
  trigger:
    include: ".child.yml"
    strategy: depend
  1. Commit a change to the a file to kick off a new pipeline (we'll call this P1)
  2. Within 5-7 seconds of creating P1, commit another change to a file to create another pipeline (we'll call this P2)
  3. Navigate to the P1 view, and you'll see that child pipelines are canceled even though their jobs don't have interruptible: true.

Analysis

As explained above, in the "Any uninterruptible jobs have not started yet." scenario, we cancel the whole pipeline. And in the customer case, if any job of a child pipeline has not started yet, we cancel it.

Desired outcome

Users want to have an option to choose if the pipeline should be canceled or not if all jobs have not started yet.

Proposal

We'll introduce a new syntax to control how auto-cancel works;

workflow:
  auto_cancel:
    on_new_commit: conservative # (default), other options: "interruptible", "none"

options:

  1. "all": when a new commit/pipeline is created, cancel all jobs.
  2. "interruptible": when a new commit/pipeline is created, cancel only interruptible:true jobs.
  3. "conservative": when a new commit/pipeline is created, behave the old/legacy way; check if there is at least one "started" interruptible:false job. If so, then don't cancel anything. If there is not, then cancel all jobs in the pipeline.
  4. "none": when a new commit/pipeline is created, do not cancel any job in the pipeline.

Users will also have these options;

Either this (A):

default:
  interruptible: true

workflow:
  auto_cancel:
    on_new_commit: interruptible
  rules:
    - if: $CI_COMMIT_REF_PROTECTED
      auto_cancel:
        on_new_commit: none

With this setting, we're keeping the interruptible of jobs but we're changing the strategy of canceling based on CI_COMMIT_REF_PROTECTED.

or this (B):

default:
  interruptible: true
  rules:
    - if: $CI_COMMIT_REF_PROTECTED
      interruptible: false

workflow:
  auto_cancel:
    on_new_commit: interruptible

With this setting, we're keeping the auto-canceling strategy but we're changing the default interruptible for jobs based on CI_COMMIT_REF_PROTECTED.

Implementation proposal
  1. Add interruptible to trigger jobs.

Currently, trigger jobs get auto-canceled if they are not started yet. However, when we implement and use on_new_commit: interruptible, they will never get auto-canceled because trigger jobs can't have the interruptible attribute. So, we also need to add the support of interruptible to trigger jobs. However, we need to have a nice documentation of this to explain how this interruptible affects trigger jobs (for example; interruptible affects non-started trigger jobs).

  1. Add CI YAML config: workflow:auto_cancel:on_new_commit.
  2. Add CI YAML config: workflow:rules:auto_cancel:on_new_commit.

Implementation table

  1. Add auto_cancel_on_new_commit to ci_pipeline_me... (!137520 - merged)
  2. Add new syntax of workflow:auto_cancel:on_new_c... (!137892 - merged)
  3. Add `interruptible` to trigger jobs (!138508 - merged)
  4. Add auto_cancel:on_new_commit to canceling redu... (!139358 - merged)
  5. workflow:rules:auto_cancel (#436467 - closed)

Follow up: workflow:rules:auto_cancel (#436467 - closed)

Edited by Furkan Ayhan