Split cron queue into priority-based queues (cron_critical, cron_standard, cron_maintenance)
Problem
The current cron
queue (weight 2) contains scheduled jobs with vastly different business priorities and impacts:
Revenue-critical jobs (should be higher priority):
-
AutoRenewCronJob
- Runs every 4 hours, directly impacts revenue -
ReconciliationCronJob
- Processes quarterly true-ups -
UpcomingReconciliationNotificationCronJob
- Time-sensitive customer notifications
Standard operational jobs (current priority seems appropriate):
-
DownloadMonthlyInvoiceCronJob
- Monthly invoice processing -
ExpireOrderTrialCronJob
- Trial expiration processing -
Cron::Zuora::SyncProductCatalogJob
- Daily catalog sync
Maintenance/audit jobs (could be lower priority):
-
Quality::TestAccountCleanupCronJob
- Test data cleanup -
Cron::Zuora::LocalCopyAuditJob
- Data consistency audits -
Cron::ErrorMonitorings::WeeklyReportJob
- Weekly reporting
Treating all these jobs with the same priority (weight 2) means that low-priority maintenance tasks can delay revenue-critical operations.
Proposal
Split the cron
queue into three priority-based queues:
cron_critical
(weight 7-8)
1. Jobs that directly impact revenue or have strict time requirements:
AutoRenewCronJob
ReconciliationCronJob
UpcomingReconciliationNotificationCronJob
- Any future time-sensitive business operations
cron_standard
(weight 4-5)
2. Regular operational jobs that should run reliably but aren't urgent:
DownloadMonthlyInvoiceCronJob
ExpireOrderTrialCronJob
Cron::Zuora::SyncProductCatalogJob
Cron::Zuora::PlanClassificationSyncJob
GitlabComSeatsUsageEmailCronJob
SelfManagedSeatsUsageEmailCronJob
SubmitSeatUtilizationDataReminderCronJob
UnlinkedSaasSubscriptionReminderCronJob
OrphanedComSubscriptionEmailCronJob
cron_maintenance
(weight 2)
3. Cleanup, audit, and reporting jobs that can run when system load is lower:
Quality::TestAccountCleanupCronJob
Cron::Zuora::LocalCopyAuditJob
Cron::ErrorMonitorings::WeeklyReportJob
AuditProvisionsCronJob
- Any future audit or cleanup jobs
Implementation Steps
- Audit all current cron jobs and categorize by business impact
- Update job classes to use new queue names:
# Before queue_as :cron # After queue_as :cron_critical # or cron_standard, or cron_maintenance
- Update
config/sidekiq.yml
to include new queues with appropriate weights - Update any monitoring/alerting that references the
cron
queue - Test in staging to ensure jobs run as expected
- Document queue selection guidelines for future cron jobs
Benefits
- Revenue-critical jobs get processed with higher priority
- Maintenance tasks don't block important operations
- Clearer intent when adding new scheduled jobs
- Better observability (can monitor critical vs. maintenance job performance separately)
Related
- Parent epic: &19587