Investigate how to improve Import & Export performance of CI pipelines

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

It's been observed that CI pipelines Import & Export is significantly slower compared to other types of relations we Export/Import. For example, a project with 7200 pipelines for one of the private projects:

1. Export of CI pipelines only took ~7 hours
2. Import of CI pipelines only took 8 hours

This of course depends on the size of the subrelations within the pipeline, but as it currently stands, for a lot of projects, this is extremely inefficient with ~15 pipelines per minute.

Here's an example visualization of a project Import from Kibana. As you can see based on logs production the majority of work happens in the first 3h and then the throughput drops significantly.

We should investigate CI pipelines both Export and Import to see if we can find a way to improve it's performance. We have made improvements to this relation in the past but it looks like we need to investigate this relation again.