[go: up one dir, main page]

Skip to content

Split cron queue into priority-based queues (cron_critical, cron_standard, cron_maintenance)

Problem

The current cron queue (weight 2) contains scheduled jobs with vastly different business priorities and impacts:

Revenue-critical jobs (should be higher priority):

  • AutoRenewCronJob - Runs every 4 hours, directly impacts revenue
  • ReconciliationCronJob - Processes quarterly true-ups
  • UpcomingReconciliationNotificationCronJob - Time-sensitive customer notifications

Standard operational jobs (current priority seems appropriate):

  • DownloadMonthlyInvoiceCronJob - Monthly invoice processing
  • ExpireOrderTrialCronJob - Trial expiration processing
  • Cron::Zuora::SyncProductCatalogJob - Daily catalog sync

Maintenance/audit jobs (could be lower priority):

  • Quality::TestAccountCleanupCronJob - Test data cleanup
  • Cron::Zuora::LocalCopyAuditJob - Data consistency audits
  • Cron::ErrorMonitorings::WeeklyReportJob - Weekly reporting

Treating all these jobs with the same priority (weight 2) means that low-priority maintenance tasks can delay revenue-critical operations.

Proposal

Split the cron queue into three priority-based queues:

1. cron_critical (weight 7-8)

Jobs that directly impact revenue or have strict time requirements:

  • AutoRenewCronJob
  • ReconciliationCronJob
  • UpcomingReconciliationNotificationCronJob
  • Any future time-sensitive business operations

2. cron_standard (weight 4-5)

Regular operational jobs that should run reliably but aren't urgent:

  • DownloadMonthlyInvoiceCronJob
  • ExpireOrderTrialCronJob
  • Cron::Zuora::SyncProductCatalogJob
  • Cron::Zuora::PlanClassificationSyncJob
  • GitlabComSeatsUsageEmailCronJob
  • SelfManagedSeatsUsageEmailCronJob
  • SubmitSeatUtilizationDataReminderCronJob
  • UnlinkedSaasSubscriptionReminderCronJob
  • OrphanedComSubscriptionEmailCronJob

3. cron_maintenance (weight 2)

Cleanup, audit, and reporting jobs that can run when system load is lower:

  • Quality::TestAccountCleanupCronJob
  • Cron::Zuora::LocalCopyAuditJob
  • Cron::ErrorMonitorings::WeeklyReportJob
  • AuditProvisionsCronJob
  • Any future audit or cleanup jobs

Implementation Steps

  1. Audit all current cron jobs and categorize by business impact
  2. Update job classes to use new queue names:
    # Before
    queue_as :cron
    
    # After
    queue_as :cron_critical  # or cron_standard, or cron_maintenance
  3. Update config/sidekiq.yml to include new queues with appropriate weights
  4. Update any monitoring/alerting that references the cron queue
  5. Test in staging to ensure jobs run as expected
  6. Document queue selection guidelines for future cron jobs

Benefits

  • Revenue-critical jobs get processed with higher priority
  • Maintenance tasks don't block important operations
  • Clearer intent when adding new scheduled jobs
  • Better observability (can monitor critical vs. maintenance job performance separately)

Related