GitLab Migration - Make it possible to parallelize pipelines
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
The problem
Some pipelines might be too time and memory consuming. Imagine a pipeline that batch loads multiple snippets, then for each snippet we clone their repository. If a project has thousands of snippets, this might mean thousands of requests to clone repos and a very long time that this job would be taking from one of the Sidekiq job workers, basically taking this worker all for itself.
The proposal
Work on the pipelines so that we can make them splittable through Sidekiq workers. This will guarantee a better distribution of the Sidekiq workers and will also allow pipelines to be executed faster, as they'll be parallelized.
Edited by 🤖 GitLab Bot 🤖