[go: up one dir, main page]

CI: Download repo from artifacts in lint stage pipelines

What does this MR do and why?

Replace git cloning with artifact-based repository fetching in lint stage to reduce Gitaly load and mitigate "too much load" errors.

This also reduces the CI build time by ~15 seconds

References

Contributes to GitLab Service Overload Impact on Pipelines ("u... (gitlab-org/quality/analytics/team#134)

Gitlab Doc link

List of jobs affected by this change

  • docs code_quality
  • docs hugo_build
  • docs-i18n-lint japanese-vale
  • docs-i18n-lint markdown
  • docs-i18n-lint paths
  • docs-lint links
  • docs-lint markdown
  • docs-lint mermaid
  • docs-lint redirects
  • docs-i18n-lint links
  • feature-flags-usage
  • generate-apollo-graphql-schema
  • haml-lint
  • lint-pipeline-yaml
  • lint-yaml
  • ping-appsec-for-sast-findings
  • qa:metadata-lint
  • qa:orchestrator-lint
  • qa:selectors
  • rubocop
  • rubocop-docs
  • check-deprecated-files
  • templates-shellcheck
  • semgrep-appsec-custom-rules
  • static-verification-with-database
  • static-analysis

JOBS That are not part of this change in STAGE Lint

  • check-deprecated-files - As clone-gitlab-repo is just cloning to depth 20 , and check-deprecated-files is not able to fetch previous commit revision Job Link

Before Logs

Getting source from Git repository
Gitaly correlation ID: b0b18ba721c14de6ab2010c6966b016f
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/gitlab-org/gitlab/.git/
Created fresh repository.
remote: Enumerating objects: 752904, done.        
remote: Counting objects: 100% (752904/752904), done.        
remote: Compressing objects: 100% (332322/332322), done.        
remote: Total 752904 (delta 535877), reused 581935 (delta 399963), pack-reused 0 (from 0)        
Receiving objects: 100% (752904/752904), 522.25 MiB | 33.48 MiB/s, done.
Resolving deltas: 100% (535877/535877), done.
From https://us-east1-c.ci-gateway.int.gprd.gitlab.net:8989/gitlab-org/gitlab
 * [new ref]               refs/pipelines/2218033785 -> refs/pipelines/2218033785
Checking out aa8eae83 as detached HEAD (ref is refs/merge-requests/214344/merge)...
Skipping Git submodules setup

After Chnage Logs

Getting source from Git repository
Skipping Git repository setup
Skipping Git checkout
Skipping Git submodules setup
Downloading artifacts
Downloading artifacts for clone-gitlab-repo (12457474851)...
Downloading artifacts from coordinator... ok        correlation_id=21a9f157dcc7490c8d897110861063eb host=storage.googleapis.com id=12457474851 responseStatus=200 OK token=6b_XKyAqB

IMPACT on CI Speed

Before After
https://pipeline-visualizer-gitlab-org-quality-engineeri-bcf92e4999c4df.gitlab.io/gitlab-org/gitlab/pipeline/2188649085 https://pipeline-visualizer-gitlab-org-quality-engineeri-bcf92e4999c4df.gitlab.io/gitlab-org/gitlab/pipeline/2219455606

gitlab-org/quality/analytics/team#134

Edited by Pranshu Sharma

Merge request reports

Loading