Draft - Direct Transfer - Allow the user to customize the number of concurrent migrated entities
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
In #387855 (closed), we began limiting the number of concurrently migrated entities per bulk import to 5.
If I remember correctly, this limit was primarily introduced to prevent excessive requests to the source instance and to reduce the number of concurrently exported projects.
However, this hardcoded cap doesn't allow Direct Transfer to process migrations quickly when the source and destination can handle multiple migrations.
For example, when migrating several small projects (fewer than 1000 MRs and CI pipelines) from GitLab.com to GitLab Dedicated, we observed that Sidekiq utilization was low due to the 5-entity limit. As a result, Sidekiq typically executed only 5 BulkImports::PipelineBatchWorker jobs concurrently.
The current workaround is to perform multiple BulkImports simultaneously, as the limit is set per BulkImport. For example, triggering two BulkImports simultaneously increases the number of concurrent entities from 5 to 10.
For a particular migration, Professional Service had to split the whole group migration into nine waves, each one handling 50 projects.
Proposed solution
One proposed solution is to allow customers to provide an extra option indicating how many entities they desire to migrate concurrently.
Since they are aware of the source destination configuration, they should know how many entities the source destination can handle
NOTE - this proposed solution is still a draft