Skip to content

Deployment · By Rivtor · Updated

Deploy a Python RQ worker with Redis

Deploy RQ as a separate Python worker process with the same job code and compatible dependencies as its producer. Connect both to the intended Redis instance and queue, budget memory and retries, and verify results after a worker restart.

See the Rivtor deployment workflow →

Package importable job functions

An RQ job refers to Python code that must exist in the worker environment. Keep the task module importable from the worker’s working directory and install the same application dependencies the task needs. A producer enqueueing a function successfully does not prove a fresh worker can import it.

For a disposable example, create tasks.py:

def add_numbers(a, b):
    return a + b

With rq and redis installed, a separate producer can enqueue it:

import os
from redis import Redis
from rq import Queue
from tasks import add_numbers
queue = Queue('reports', connection=Redis.from_url(os.environ['REDIS_URL']))
job = queue.enqueue(add_numbers, 2, 3, job_timeout=60)
print(job.id)

This example writes a synthetic job to the selected Redis instance. Do not point it at a customer queue merely to test configuration.

Start the correct queue consumer

Use RQ’s current worker configuration to supply REDIS_URL privately and select reports as the queue. Avoid placing credentials in shell history or printed command examples. Check the pinned RQ version’s supported connection configuration.

Rivtor documents RQ worker detection. A worker should be a private process without a public application URL. Inspect the detected command rather than adding a dummy web server solely to make a worker look healthy.

Budget retries and retained data

Choose job timeout, retry policy and result retention for the workload. A long job can occupy a worker while other jobs wait. Large results retained in Redis consume the same memory budget as queued work. Failure retention is useful for investigation but still needs a limit.

Verify the selected Redis memory and persistence policy. Caching-oriented eviction can remove queue data; managed hosting does not automatically make every Redis-backed job durable.

Observe a controlled restart

Enqueue the example and confirm the expected result is 5. Then use a deliberately slow synthetic task to observe shutdown and restart behaviour. Check whether work completes, fails or is retried, and whether side effects can repeat. Use idempotent output keys or database constraints where appropriate. A worker process running with an empty log is not evidence that it is consuming the intended queue.

References