Move hardcoded number of import jobs limit to a application setting
We have noticed that the current import jobs limit of 1000 on #432137 (closed) may slow imports. This is because the number is insufficient to allow the jobs to be processed at the same rate they are enqueued. As a result, jobs may be scheduled too far in the future.
Although we won't be increasing the number on GitLab.com, allowing the limit to be configured will allow Dedicated and SM administrator to configure the number according to their needs.
Proposed solution
Create a new application settings to store the limit.
The default value of the settings should be:
- GitHub Importer: 1000
- BitBucketCloud: 100
- BitBucketServer: 100
GitHub Import
The hardcoded limit is defined here
Please, also remove the github_import_increased_concurrent_workers feature flag introduced to test an increase limit.
Note: The pull requests stage overrides the limit and is set to 200. For now, let's keep the limit to 200 in this stage.
BitBucketServer
The hardcoded limit is defined here
BitBucketCloud
The hardcoded limit is defined here
Documentation
Explain in the docs what the limit does.