v2.9 · Production runbook

Queue Backlog Recovery Runbook

Recover from a growing outbound queue without causing a second incident.

Stabilize

Use timestamps, queue IDs, remote responses, source IP, VMTA, sender domain and message class to establish the failure boundary. Avoid reacting to one isolated delivery attempt.

Stop uncontrolled growth before increasing throughput. Identify whether backlog is caused by injection, local processing, DNS, connectivity, provider deferrals or insufficient workers.

Metrics

  • Queue depth and oldest age
  • New injection rate versus successful delivery rate
  • Deferral distribution by domain and VMTA
  • CPU, memory, disk latency and file descriptors

Drain plan

  1. Throttle or pause new campaigns.
  2. Protect transactional traffic.
  3. Fix the limiting dependency.
  4. Increase throughput in measured steps.
  5. Confirm backlog age is falling continuously.
Operational rule: collect evidence before changing policy, make one controlled change, verify the result, and retain a rollback path.
Search Trushilla Documentation