Skip to content

Architecture · By Rivtor · Updated

Graceful shutdown for Node.js and Python background workers

On shutdown, stop taking new jobs, handle in-flight work through the queue library’s supported lifecycle and close connections within the host’s grace period. Design for forced interruption as well, because graceful shutdown cannot guarantee every task finishes.

See the Rivtor deployment workflow →

Treat shutdown as a job-state transition

A rolling release, operator restart or resource event can stop a worker while it owns a task. The application must distinguish a completed side effect from an acknowledged message. If a process dies between those actions, the next worker may need to replay work whose effect already happened.

Use the queue library’s shutdown API rather than exiting immediately in a generic signal handler. Check the installed version’s semantics and the host’s termination window. A graceful operation that waits forever can still be interrupted by the platform.

Define the sequence

StageIntended behaviour
Shutdown requestedStop accepting additional work
Active jobsComplete or become safely recoverable
Deadline reachedRecord what remains uncertain
CleanupClose broker/database clients and temporary files
RestartReconcile recovered jobs with durable outputs

For Node.js, keep the event loop free enough to process the shutdown signal. For Python process pools, account for how child tasks receive termination and how acknowledgements are configured. Avoid assuming the same signal handling applies to every worker library.

Make interruption safe

Use idempotent operations, durable task identities and uniqueness checks where appropriate. Keep database transactions short and do not hold one open across an unbounded external API call. If an external provider supports idempotency keys, use them for repeatable requests.

Rivtor documents private worker services. Verify its current lifecycle and grace behaviour for the actual workload; this guide does not establish a universal shutdown guarantee. Check Redis policy separately from process shutdown.

Run two different drills

First, request a normal shutdown during a synthetic task and observe whether it finishes before exit. Second, interrupt a disposable worker abruptly and confirm recovery behaviour. Compare job identity, durable output and duplicate effects.

Check that a shutdown error is visible without printing credentials or customer payloads. Record queue version, task duration and host deadline with the result. Passing the graceful drill alone is insufficient because a memory kill or machine failure may provide no cleanup opportunity.

References