Adjust queue weights to separate user-facing vs internal operations

Problem

Some queues currently have weights that don't accurately reflect their impact on customer experience. Specifically:

Under-prioritized (customer-facing, should be higher):

• expiration (weight 3): Trial expirations are time-sensitive and directly impact customer experience. When a trial expires, customers expect immediate changes to their access.

Over-prioritized (internal operations, could be lower):

• zuora_product_sync (weight 3): Product catalog synchronization is important for data consistency but doesn't directly block customer operations. It runs daily and doesn't need to compete with customer-facing jobs.

Proposal

Adjust weights to better reflect customer impact:

Increase Priority for Customer-Facing Operations

expiration queue: 3 → 7-8

• Handles trial expiration processing via ExpireOrderTrialJob
• Time-sensitive: Customers expect trials to expire at the specified time
• Directly impacts customer access and experience
• Should be processed quickly to maintain trust and meet expectations

Decrease Priority for Internal Sync Operations

zuora_product_sync queue: 3 → 4-5

• Handles Zuora::SyncProductCatalogJob (runs daily at midnight)
• Internal data synchronization, not customer-blocking
• Can tolerate some delay without customer impact
• Should not compete with customer-facing provisioning or expiration jobs

Additional Considerations

While making these changes, we should also review:

1. reconciliations queue (weight 3): Contains both customer-facing notifications and internal processing. Consider if this needs further splitting.

2. auto_renewals queue (weight 3): Revenue-critical operations that might warrant higher priority (weight 6-7).

3. workato queue (weight 3): Lead creation workflows - assess if these are customer-blocking or can be slightly lower priority.

Implementation Steps

1. Update config/sidekiq.yml:

   :queues:
     # ... other queues ...
     - [expiration, 7]  # increased from 3
     # ... other queues ...
     - [zuora_product_sync, 4]  # decreased from 3

2. Monitor queue metrics before and after the change:

   • Queue depth
   • Job latency (time from enqueue to execution)
   • Job duration
   • Customer-reported issues related to trial expirations

3. Validate in staging environment first

4. Document the rationale for these weights

Benefits

• Trial expirations processed more reliably and quickly
• Better customer experience during time-sensitive operations
• Internal sync operations don't compete with customer-facing jobs
• More accurate reflection of business priorities in queue configuration

Related

• Parent epic: &19587