[go: up one dir, main page]
Closed
Milestone Apr 24, 2023–Jun 16, 2023

(2023Q2) Pipeline Monitoring, Profiling and Simulation

snipped from https://hackmd.io/vdYP7DO5RDatOwG686ez3Q

Title : Pipeline-centric Infrastructure Monitoring

Description : Given the OKR, we need tools to monitor and measure the pipeline performance. We also need to tools to help us evaluate the changes we propose.

Objective : Establish metrics to measure progress on the Axes 3 & 4.

Estimated effort (in nb weeks) : --

Associated KRs : CI runs consistently under 20 minutes and cost is not higher that the current one

Dependencies : None

Deliverables : The set of metrics necessary to measure progress on the Axes 3 & 4.

Part of : https://gitlab.com/groups/tezos/-/milestones/5

Task breakdown

  • Define pipeline types (@arvidnl)
  • Graph flakiness over time (@arvidnl)
  • Develop a pipeline profiler allowing us to estimate pipeline performance improvements w.r.t. a given baseline (@abate)
    • Add pipeline type 'before-merging' ((@abate)
    • Add pipeline type 'before-merging w/ opam' ((@abate)
    • Add pipeline type 'before-merging marge-bot w/ opam' ((@abate)
    • Add pipeline type 'before-merging marge-bot ((@abate)
    • Add pipeline type 'scheduled' ((@abate)
  • Pipeline monitoring dashboard as proposed on #nl-devops by Arvid (@abate)
  • Profile all jobs to understand resources requirements (charles, corentin)
  • (Pipeline cost estimation tool)

Deliverables

The deliverables for this project will be two tools producing metrics necessary to measure progress on Project 3 & 4.

  • Tool: Pipeline simulator (M1)
    • For a set of pipeline types, disregarding evolutions in the code-base unrelated to our changes.
      • Metric M1.1: Projected Pipeline wall-time and sequential duration by type
      • Metric M1.2: Projected Total Sequential Time (obtained by multiplying M1.1 with pipeline type frequencies)
      • Metric M1.3: Projected Sequential Time per Pipeline (obtained by dividing M1.2 with sum of pipeline type frequencies)
  • Tool: Pipeline monitoring dashboard (M2)
    • For a given project & given time period & pipeline type:
      • Metric M2.1: Recorded Pipeline average/worst-case wall-time and sequential duration
      • Metric M2.2: Recorded Number of launched pipelines for a given project
      • Metric M2.3: Recorded Sequential Time per Pipeline
  • Report with the resource characterization of each job in the set of pipeline types defined in project 1.

Figure out a way to measure progress modulo evolution

Breakdown of task 1
  1. Identify a set of pipeline types which we would like to measure
    • assignee: arvid
    • estimation: 1d
    • description: types & (frequence sur period donnée)
  2. Make a mechanism for running the set of pipelines for a given reference commit + a set of configuration changes
    • Il faut un infrastructure apart?
      • assignee: charles
      • estimation: 1d
      • description:
    • outil: input: baseline commit + modificiations commits
      • assignee: pietro?
      • estimation: 5d
      • checkout baseline commit
      • cherry pick modifications commits
      • modifier la CI config (.gitlab-ci.yml) automatiquement pour ajotuer un tag qui correspond à l'infra dedié
        • .gitlab-ci.yml: default: tag: MACHIN
        • comment gerer arm64?
      • lancer les pipelines types
        • instrumentaiton de .gitlab-ci.yml necessaire pour simuler des pipelines types
      • dans un projet apart (nomadic-labs/tezos-ci-measures)
  3. (Make it possible to do this every week, monitor the result and synthethize)

Pipeline monitoring dashboard

Ref: pipeline monitoring dashboard