Bitbucket Server Importer - Increase concurrent_bitbucket_server_import_jobs_limit
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Introduced in #451129 (closed), the feature flag bitbucket_server_notes_separate_worker is now enabled by default (#456262 (closed)).
With this change, Bitbucket Server Importer now imports individual records as separate jobs, rather than grouping them into fewer jobs that import multiple records.
As a result, the concurrency limit for importer jobs (currently set to 100) needs to be increased.
Now that Bitbucket Server Importer works similarly to GitHub Import, the concurrency may match GitHub Import's import job limit, which is 1000.
To test if the new limit is safe, we will release it behind a feature flag and gradually roll it out.
More context about #451129 (closed)
Previously, the importer queued multiple jobs that made simultaneous API calls to Bitbucket Server to fetch the notes of each pull request and then imported the notes returned by the API. To reduce the load on both Bitbucket Server from the concurrent requests and the load on the GitLab database, a low limit was set (100 jobs per minute). Note that with this approach, the number of importer notes per minute was unpredictable, as each job could import many notes returned by the API.
With the introduction of #451129 (closed), the importer now has one worker that makes sequential API calls to fetch the note details of all pull requests, and for each fetched note, an individual job is enqueued and scheduled to run over time, as defined by the concurrent_github_import_jobs_limit. For example, if the limit is set to 100, 100 jobs will be spread to run over 1 minute. However, that means only 6000 notes will be imported per hour, which is a low number that delays the migration.