Create dedicated queue for time-sensitive notifications

Problem

Currently, notification jobs are scattered across multiple queues with varying priorities:

Existing notification queues:

• monthly_notification (weight 2): Monthly seat usage digests
• mailers (weight 2): General email delivery
• Various queues contain notification jobs mixed with other operations

Issues with current setup:

1. Time-sensitive notifications (trial expiration, payment failures) compete with scheduled digests
2. No clear distinction between urgent vs. routine notifications
3. Difficult to monitor and optimize notification delivery separately

Examples of time-sensitive notifications:

• Trial expiration warnings (should be immediate)
• Payment failure alerts (revenue-impacting)
• License delivery notifications (customer-blocking)
• Reconciliation alerts (time-bound)

Proposal

Create a dedicated high-priority queue for urgent, time-sensitive notifications.

New Queue Structure

notifications_urgent (weight 7-8) Purpose: Time-sensitive, customer-impacting notifications that should be delivered immediately

Jobs to move here:

• License delivery notifications (from gitlab queue)
• Trial expiration warnings (from expiration queue)
• Payment failure alerts
• Reconciliation deadline notifications
• Any notification where delay impacts customer experience

Keep monthly_notification (weight 3) Purpose: Scheduled digest emails that can tolerate some delay

Jobs that stay here:

• SeatsUsageMonthlyNotificationJob
• Monthly usage reports
• Periodic summary emails

Keep mailers (weight 3) Purpose: General transactional emails

Jobs that stay here:

• Order confirmations
• Account updates
• General customer communications

Benefits

1. Better customer experience: Urgent notifications delivered promptly
2. Clear prioritization: Explicit distinction between urgent and routine
3. Improved monitoring: Can track urgent notification delivery separately
4. Reduced risk: Payment and expiration alerts won't be delayed by digest emails

Implementation Steps

1. Identify all notification jobs:

   • Audit codebase for mailer jobs
   • Categorize by urgency and customer impact
   • Document current queue assignments

2. Create new queue:

   # app/jobs/notifications/urgent_base_job.rb
   module Notifications
     class UrgentBaseJob < ApplicationJob
       queue_as :notifications_urgent
       # Common configuration for urgent notifications
     end
   end

3. Update job classes:

   # Example: License notification
   class LicenseNotificationJob < Notifications::UrgentBaseJob
     def perform(subscription_id)
       # Send license email
     end
   end

4. Update config/sidekiq.yml:

   :queues:
     # ... other queues ...
     - [notifications_urgent, 8]
     # ... other queues ...
     - [monthly_notification, 3]
     - [mailers, 3]

5. Add monitoring:

   • Alert on queue depth for notifications_urgent
   • Track delivery latency for urgent notifications
   • Monitor failure rates

6. Document guidelines:

   • When to use notifications_urgent vs. mailers
   • SLA expectations for each notification queue
   • Examples of urgent vs. routine notifications

Examples of Queue Assignment

Use notifications_urgent:

• ✅ Trial expires in 24 hours
• ✅ Payment method failed
• ✅ License ready for download
• ✅ Reconciliation due in 7 days
• ✅ Subscription cancelled (immediate impact)

Use monthly_notification:

• ✅ Monthly seat usage digest
• ✅ Quarterly usage summary
• ✅ Monthly invoice available

Use mailers:

• ✅ Order confirmation
• ✅ Account settings changed
• ✅ Password reset
• ✅ General announcements

Success Criteria

• All time-sensitive notifications identified and moved to new queue
• Urgent notifications delivered within 1 minute of enqueue (P95)
• No customer complaints about delayed critical notifications
• Clear documentation for future notification jobs

Related

• Parent epic: &19587
• Related: #14270 (user-facing operations)