Skip to content

Choosing a platform · By Rivtor · Updated

Background worker hosting: queues, retries and always-on compute

A background worker needs a process that can keep consuming jobs, a reachable broker, bounded concurrency and a safe restart strategy. Compare worker compute and queue durability separately from web hosting, then test retries and shutdown with representative jobs.

Compare Rivtor plans for your workload →

Separate the queue from the process

The queue stores pending work; the worker executes it. Hosting a Redis instance does not mean a worker is running, and keeping a web server awake does not guarantee jobs are consumed. Inventory the producer, broker, worker and output store before comparing providers.

Consider a report that takes 40 seconds to generate. The API can return a job identifier while a worker processes the report. The browser polls an authorized status route. That design avoids keeping one HTTP request open, but it introduces retry, ownership and durability decisions that a hosting button cannot resolve.

Compare the operational requirements

RequirementAcceptance check
Continuous executionWorker consumes a job after the website has been idle
Safe retriesRepeated job delivery does not duplicate the final effect
Resource fitConcurrency stays within memory and connection limits
Broker persistencePending-job behaviour matches your loss tolerance
ShutdownIn-flight work completes or becomes safely retryable

Sleeping compute can be suitable for interactive prototypes but should not be assumed suitable for a worker that must react immediately. Check the selected plan’s actual worker and idle behaviour. Compare compute, broker, retained job data and storage together.

Build a restart drill

Use synthetic jobs with known inputs. Enqueue a small batch, stop the worker midway, restart it and inspect completed, failed and waiting counts. A queue dashboard showing zero waiting jobs is insufficient if jobs were lost or outputs were written twice. Compare unique job identities to durable results.

Make expensive external side effects idempotent where possible. Put a maximum on attempts and use backoff so an unavailable dependency does not cause a retry storm. Failed jobs need a review and recovery process.

Evaluate Rivtor for this workload

Rivtor documents Celery, BullMQ and RQ detection and private worker services. Review framework configuration and Redis limits before selecting a plan. A documented Redis memory policy may be suitable for caching but unsuitable for a durable queue; verify it instead of assuming managed Redis means durable jobs.

References