Move GPG processes to background jobs
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
This is follow up work from #20918 (closed) where there is a lot of good technical information.
If GPGME is unable to cleanup the homedir for any reason (directory doesn't exist, directory not empty), the gpg-agent process will remain running.
If we move the spawn of any gpg process away from the web-nodes and into Sidekiq, we'll be able to better control how we handle the processes. We can wait until sidekiq has been able to delete the homedir so that the gpg-agent can kill itself. Currently, we are retrying - which does help reduce the processes, but can still leave processes lingering when unsuccessful or when we do not retry enough.
More information on that here and here.
Steps to reproduce
Potential steps to reproduce are listed here, however, it is hard to reproduce this as the homedir may get cleaned up before you have time to check anything.
Relevant logs and/or screenshots
Associated issue has plenty of logs in the comments.
https://gitlab.com/gitlab-org/gitlab/blob/33780dd6ccf233827e9e49967c391c1373c4a83c/lib/gitlab/gpg.rb#L7-9 lists information on the timeout for Rails.