Add purge cache functionality for Maven virtual registry upstreams

What does this MR do and why?

In Maven virtual registry MVC (API only interactions) (&14137 - closed), there is no way to clear the entire cache of a given upstream.

In this MR, we add a new rest API endpoint that purges the entire cache of an upstream:

• DELETE /api/v4/virtual_registries/packages/maven/upstreams/:id/cache

How cache entries for an upstream are purged?

• An upstream can have a large no. of cache entries, and marking all of them for destruction during the web request could lead to timeouts. That's why a background worker is introduced to batch over an upstream cache entries and mark each batch for destruction.

• Since cache entries are associated with external objects (files on the Object Storage), we update their status as pending_destruction, and then a background worker will pick them up and execute the actual deletion.

References

• Virtual registry: clear upstream cache (#538327 - closed) • Moaz Khalifa • 18.2 • Needs attention

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Enable the feature flag : Feature.enable(:maven_virtual_registry)
2. Have a PAT (scope api) and a GitLab instance with an EE licence ready.
3. In rails console:

   # stub file upload
   def fixture_file_upload(*args, **kwargs)
     Rack::Test::UploadedFile.new(*args, **kwargs)
   end
   
   # create a maven upstream
   u = FactoryBot.create(:virtual_registries_packages_maven_upstream)
   
   # create a few cache entries that belong to the upstream
   cache_entries = FactoryBot.create_list(virtual_registries_packages_maven_cache_entry, 3, upstream: u)

4. Clear the upstream cache:

   curl -v -X DELETE -H "PRIVATE-TOKEN: PAT" "http://gdk.test:3000/api/v4/virtual_registries/packages/maven/upstreams/<u.id>/cache"

5. In rails console, make sure that all cache entries have been marked for destruction:

   u.default_cache_entries
   => []
   
   u.cache_entries.pending_destruction.size
   => 3

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #538327 (closed)