Cleanup before Gitlab::SidekiqMiddleware::MemoryKiller acts on project import workers
Right now if the sidekiq cluster goes above a certain memory level MemoryKiller
will send a SIGTERM
to the workers in that cluster.
After 15 minutes it will send a SIGKILL
effectively killing every worker and without letting a cleanup happen first.
That worker is then reset and has the opportunity to run again, but, because the import job did not change it's state to failed
(because it was killed without proper cleanup) the retried worker will think that there is already another worker processing that import job and it will return silently.
Proposed solution
When we want to effectively kill the worker we go through https://github.com/mperham/sidekiq/blob/7fe2515138177eea88823398423470834806b323/lib/sidekiq/processor.rb#L60
We would be able to rescue Sidekiq::Shutdown
from within the worker, cleanup (set the mirror to failed) and re-raise so that the worker can be retried gracefully.
\cc @pcarranza @DouweM