[go: up one dir, main page]

Skip to content

Cleanup before Gitlab::SidekiqMiddleware::MemoryKiller acts on project import workers

Right now if the sidekiq cluster goes above a certain memory level MemoryKiller will send a SIGTERM to the workers in that cluster.

After 15 minutes it will send a SIGKILL effectively killing every worker and without letting a cleanup happen first.

That worker is then reset and has the opportunity to run again, but, because the import job did not change it's state to failed (because it was killed without proper cleanup) the retried worker will think that there is already another worker processing that import job and it will return silently.

Proposed solution

When we want to effectively kill the worker we go through https://github.com/mperham/sidekiq/blob/7fe2515138177eea88823398423470834806b323/lib/sidekiq/processor.rb#L60

We would be able to rescue Sidekiq::Shutdown from within the worker, cleanup (set the mirror to failed) and re-raise so that the worker can be retried gracefully.

\cc @pcarranza @DouweM