Increase Sidekiq queue weight granularity from 1-4 to 1-10 scale

Problem

Currently, our Sidekiq queue weights use a limited 1-4 scale, which provides only coarse-grained control over job prioritization. As our application has grown and queue complexity has increased, this limited range makes it difficult to:

• Differentiate between critical vs. important jobs within the same weight tier
• Add new queues without disrupting existing priority relationships
• Fine-tune performance based on observed queue behavior

Proposal

Expand the weight scale from 1-4 to 1-10 to provide finer control over queue prioritization.

Benefits

1. Better differentiation: More granular weights allow us to express nuanced priority differences
2. Future flexibility: Room to add new queues without restructuring existing weights
3. Clearer intent: A 1-10 scale makes relative priorities more obvious (e.g., weight 10 is clearly critical, weight 5 is medium)
4. Easier adjustments: Can make incremental priority changes without major restructuring

Example Mapping

Current → Proposed:

• Weight 4 (Critical) → Weights 8-10
  • health_check: 10 (system stability)
  • gitlab, zuora_callback: 9 (customer-facing provisioning)
  • zuora : 8 (Zuora local copy updates)
• Weight 3 (Important) → Weights 5-7
  • expiration, salesforce: 7 (time-sensitive customer impact)
  • reconciliations, auto_renewals: 6 (revenue operations)
  • default, workato, zuora_product_sync: 5 (general operations)
• Weight 2 (Medium) → Weights 3-4
  • cron, usage_billing: 4 (scheduled operations)
  • mailers, monthly_notification: 3 (notifications)
• Weight 1 (Low) → Weights 1-2
  • action_mailbox_routing, action_mailbox_incineration: 1 (lowest priority)

Implementation Steps

1. Document current queue behavior and metrics
2. Propose new weight assignments based on business criticality and job characteristics
3. Update config/sidekiq.yml with new weights
4. Test in staging environment
5. Monitor queue performance after deployment
6. Document the new weight scale and guidelines for future queue additions

Related

• Parent epic: &19587