Respect the rate limiter for the Manifest imports

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Problem

The Manifest file importer seems to exceed the rate limits when importing many files (i.e. 866) both locally and on GitLab.com, as clicking the green Import 886 repositories button will trigger 886 HTTP requests to start importing projects. The rate limiter is defined in the superclass of the Manifest controller.

This behavior causes the imports that exceed the limit to fail without letting the user know exactly what happened. The correct behavior would be for the user to be able to successfully import a large number of repos from the manifest by clicking the Import All button. The limits should still be respected and this feature should not allow the user to circumvent the rate limitations. So, all repos would be imported, but at a throttled speed.

Proposed solution

1. Create a new endpoint for the Import All, change the Import All button to call the new action;
2. The new action can get all the repositories from Gitlab::ManifestImport::Metadata, which was created on the upload action, and enqueue their import using the ApplicationWorker.bulk_perform_in;

   ManifestImportWorker.bulk_perform_in(
     0,                      # no delays to start
     repository_params,      # A list of git remotes and destination full path from the manifest
     batch_size: 6,          # 6 jobs enqueued per time
     batch_delay: 1.minute   # time to wait before enqueueing next batch
   )

3. Within the import job we should still increment the throttling key for the current user, to limit the user to user other importers;