Skip to content

AI and agents · By Rivtor · Updated

Deploy an AI document-processing worker with durable job state

Use an API to accept authorized uploads, object storage for durable files, a queue for work and a separate worker for document processing. Store job status and output references durably so restarts and retries do not lose or duplicate the customer’s result.

See the Rivtor deployment workflow →

Pass references instead of large documents

The API should validate an upload, store it in an authorized object location and enqueue a job containing an internal identifier. Avoid embedding the whole document or a long-lived credential in the queue payload. A private object reference still needs an ownership check before a worker or browser can access it.

Consider an illustrative invoice-extraction workflow. The API creates a pending job row, stores a small test PDF and enqueues its identifier. A worker extracts text and calls an external model service. The application host runs the pipeline; it does not imply local GPU inference or a specific document-parser package is supported.

Make status transitions observable

StateMeaning
PendingDurable request exists but work has not started
RunningWorker has claimed the task
CompletedOutput reference is committed and accessible to its owner
FailedError class and recovery decision are recorded

Use a unique job identity so duplicate delivery does not create multiple final records. A crash can occur after an external API call but before writing completion; decide how that uncertainty is handled. Do not promise exactly-once execution from an ordinary queue.

Bound resources and retention

Set file-size, page-count, execution-time and concurrency limits. PDF and image parsing can consume substantially more memory than a small text request. Keep temporary files bounded and delete them when work finishes; durable input and output belong in appropriate storage.

Review Rivtor worker support, data limits and the broker’s memory policy. A caching-oriented Redis policy may not meet job durability requirements.

Rehearse interruption with synthetic files

Submit one supported document, an oversized file and a malformed file. Stop a worker during processing and observe whether the job becomes retryable. Confirm the final output belongs to the submitting account, and that a failed job can be investigated without revealing document contents in public logs. Compare queue identifiers, durable status and output counts rather than relying solely on a successful worker exit.

References