[go: up one dir, main page]
Skip to content

GitHub AdvanceStageWorker should use github_importer queue namespace

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

This issue came out of this discussion: !138437 (comment 1693841950)).

About

GithubImport::AdvanceStageWorker is the only worker in the GithubImport namespace that doesn't share the github_importer Sidekiq queue namespace.

Our docs explain the purpose of queue namespaces:

Defining a queue namespace for a worker makes it possible to start a Sidekiq process that automatically handles jobs for all workers in that namespace, without needing to explicitly list all their queue names. If, for example, all workers that are managed by sidekiq-cron use the cronjob queue namespace, we can spin up a Sidekiq process specifically for these kinds of scheduled jobs. If a new worker using the cronjob namespace is added later on, the Sidekiq process also picks up jobs for that worker (after having been restarted), without the need to change any configuration.

Problem

In !138437 (merged) we initially made a change to put GithubImport::AdvanceStageWorker in the github_importer queue namespace. It added a post-migrate migration to move jobs from the old queue to the new queue.

It prompted a discussion about the time it might take between a prod deploy and the post-migration !138437 (comment 1693841950):

I'm trying to work out how long a job queued to github_import_advance_stage might be stuck until it's moved to github_importer:github_import_advance_stage and processed [...] Greater than a day can lead to a GitHub import failing because a StuckProjectImportJobsWorker cron worker will come and fail the import after 24 hours of no activity.

We decided to go ahead with the other changes, and go back later and consider the task of changing namespaces for this worker.

Questions for refinement

  1. What is the risk of not having AdvanceStageWorker share the github_importer queue namespace with the other workers, now and in the future?
  2. Can we ensure that we can migrate jobs in a way where running GitHub imports will not fail?

Reading:

Proposal

  1. Migrate GithubImport::AdvanceStageWorker to github_importer Sidekiq queue - ideally with no possibility of failing in-progress GitHub imports.
  2. Have the worker include GithubImport::Queue which will add it to the github_importer queue.
Edited by 🤖 GitLab Bot 🤖