Sticky Runners MVC: Ability to specify that all jobs in a pipeline execute on a single GitLab Runner. (shell executor)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem

The user Job to be Done (JTBD), or problem to solve, is that their CI/CD pipelines execute reliably and in as short a time as possible. Today, a significant amount of pipeline execution time is due to the uploading and downloading of intermediate build elements between jobs in a pipeline. This is especially true for pipelines with multiple jobs that create intermediate build elements GB's in size.

The outcome is to have a solution that allows for source code, dependencies, and compiled binaries to be persisted and shared between pipeline stages without the overhead of archiving, uploads, and downloads of files.

Overview

The MVC Sticky Runner feature will enable a user to specify that all jobs in a pipeline execute on a single GitLab Runner configured to use the shell executor. The goal is to implement a MVC solution that simplifies the passing of intermediate build data between stages in a pipeline. As a result, users can easily configure and execute on a single Runner, simple multi-stage pipelines that do not require parallel job execution. Users will no longer need to rely on artifacts passing between stages to address use cases addressed by this feature.

Intended use case

• For the MVC, the primary use case that we are targeting is a pipeline that generates intermediate (temporary) build files, sometimes GBs in size, typically in the first pipeline stage and that needs to be used in jobs in subsequent stages of the same pipeline.

Example use cases/pipeline workflows

Android app

# Example Use Case Scenario: Android project - Gradle build (Check and assemble stages depends on files created in the build stage. For example - .class files)
# The .class files are only complied once. This is in the build job in the build stage.
# In this use case the .class files need to be passed efficiently from build to check and to assemble stages.

stages:
  - build
  - check
  - assemble

build_job:
  stage: build
  script:
    - echo "Build the application by simply running the Gradle tasks that will compile the .java files into .class files."
    - echo ".class files required in the test job in the check stage."

test_job:
  stage: check
  script:
    - echo "This job runs a test suite. For example - run static code analysis tools like Findbugs."
    - echo ".class files required in the assembe stage in order to assemple fhe final .apk file"

assemble_job:
  stage: assemble
  script:
    - echo "Use .class files to assemble the final .apk file. “

Node modules

stages:
  - setup
  - build
  - test
  - deploy

setup_job:
  stage: setup
  cache:
    paths:
      - node_modules/
      - bower_components/
  script:
    - nvm install version x
    - node -v
    - npm update -g npm
    - npm -v
    - npm install bower
    - npm install
    - node_modules/.bin/bower update --force-latest --allow-root

test_job:
  stage: test
  cache:
    paths:
      - node_modules/
      - bower_components/
  script:
    - echo "This job runs a test suite. .”

Proposal

• Stickiness will be configured in the .gitlab.ci YAML file for the entire pipeline by specifying an option similar to default: workspace: shared This would mean that all jobs inside of this .gitlab-ci.yml are going to be using the same Runner.
• Implement code to make the StuckCiJobsWorker aware of the Sticky runner configuration and give it different timeouts for jobs that are in such pipeline.
• When the first job of the pipeline is started and sticky Runners are on, GitLab saves the ID of the Runner that received the job. GitLab CI will not assign jobs other than from the same pipeline to this Runner.
• GitLab will send information in the job payload that the job is part of a sticky pipeline.

Example pipeline for MVC

The workspace:shared definition at the top of the .gitlab-ci.yml file informs GitLab CI that all of the jobs inside of this pipeline will be executed on the same Runner.

default:
    workspace: shared

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - This job builds something.
    - This job will generate files that are required for the test_job in the test stage.

test_job:
  stage: test
  script:
    - This job runs a test suite. It will only run when the build_job in the build stage is complete."

deploy_job:
  stage: deploy
  script:
    - echo This job will deploy the code to a target environment.

Implementation - coding tasks

• GitLab: Introduce the workspace: shared keyword, send some information to the Runner that a workspace is going to be used. (Development task for the RAILS code base).
• GitLab: Have GitLab save the runner information with the pipeline if workspace: shared is defined so that it picks up that runner again for the next job. (Development task for the RAILS code base).
• Runner: Look for the workspace information for the shell executor, and if it's defined generate the directory to be identified by the pipelineID only and not with the concurrencyID.

Not doing for MVC

• Docker executor
• Kubernetes executor
• SSH executor
• Custom executor
• Docker Machine
• Parallels
• VirtualBox

Other notes

• When Sticky Runners are enabled, it will be up to the user to decide if to run parallel jobs on the same environment. To prevent parallelization the user can use configuration options that are already available: set concurrent = 1 on the Runner or use resource_groups.

Questions

Question	response
Should runners process only jobs from a specific pipeline once they've started one job from that pipeline?	tbd
How will workspaces work for jobs with manual deployments?	tbd

Future work/iterations

• Enable sticky runners for jobs in the pipeline.
• Automatically set git strategy to none for jobs in a sticky pipeline.
• Introduce sticky runners to other executors based on customer feedback.

User Requirements (summarized)

• It is essential to have runners 'sticking' to all job stages and persistent workspace on a particular runner machine.
• I want folders like vendor/ updated, installed, built once and then re-used in following pipeline stages without having a lot of these folders stored as artifacts.
• For my Android project I need the .class files that are created in the first pipeline stage to be passed to subsequent stages of the same pipeline in a guaranteed manner and without needing to upload and download the files from GitLab.
• I need an efficient way to use the same Runner for subsequent jobs of the same pipeline execution.
• I need a feature that allows me to keep temporary build data between stages. This needs to be reliable and not interfere with current GitLab Runner features.
• I need a solution that is able to solve for anything with a compilation step, (example Java, Javascript, Android) where having some sort of temporary build data is vital.
• I want separate stages and separate containers per stage, but the ability to say that these are all tightly integrated steps of a single build path and should run on a single Runner and share files directly.
• I want a feature that I can use for my NuGet build pipeline so I am not constantly required to do a .net build for each job in the pipeline.
• I need an option that allow me to force execution of the entire pipeline on the same Runner the first job in the pipeline started on.
• I need a way of making files available between jobs/stages without having to store them as artifacts in the GitLab instance.
• I need to be able to have a simple pipeline with several stages run on a single runner without wasting time and bandwidth copying files between stages/jobs.

Reference: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/336

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.