Troubleshooting · By Rivtor · Updated
Diagnose BullMQ stalled jobs after a restart
A BullMQ job can become stalled when the worker no longer maintains its lock, including during a crash or event-loop blockage. Investigate worker lifecycle, CPU work and Redis connectivity, then verify idempotent recovery instead of deleting queue state.
See the Rivtor deployment workflow →
Understand the lock boundary
The worker uses a lock while processing a job. If the lock is not renewed, BullMQ can classify the job as stalled and recover it according to its settings. This means a task can run again after an interruption; application side effects must account for repeated execution.
Do not equate a stalled event with a task that merely runs for a long time. Inspect timestamps, worker restarts, event-loop blocking and connection failures near the event.
Investigate the likely causes
| Evidence | Investigation |
|---|---|
| Worker terminated during deploy | Shutdown handling and grace period |
| CPU-heavy synchronous code | Event-loop delay and processor isolation |
| Redis connection errors | Network route, limits and reconnect behaviour |
| Missing expected result | Whether side effects completed before interruption |
Check the installed BullMQ version’s settings and events. Do not add obsolete scheduler components or lock settings copied from a different major version without confirming they apply.
Make retries safe before replaying work
Use a durable job identity or application uniqueness constraint to prevent duplicate final records. For external API calls, use the provider’s idempotency mechanism where available. A worker can complete an external side effect and crash before recording completion, so queue state alone cannot tell you whether repeating the action is safe.
Never clear Redis, flush the database or delete the queue as a routine fix. Preserve diagnostic state and isolate a synthetic reproduction. Check Rivtor’s Redis limits; an eviction policy suitable for caches can conflict with durable job requirements.
Rehearse shutdown and restart
Run a synthetic job with observable checkpoints. Request a normal worker shutdown and confirm it stops accepting new work while handling active work according to the library’s supported lifecycle. Then test an abrupt interruption in a disposable environment and inspect the eventual output.
Measure event-loop delay and resource use under representative load. Reducing concurrency or isolating CPU work can be more appropriate than simply increasing lock duration. Keep recovery observations separate from a claim of exactly-once processing, which an ordinary queue does not establish.