AI and agents · By Rivtor · Updated
Deploy an AI document-processing worker with durable job state
Use an API to accept authorized uploads, object storage for durable files, a queue for work and a separate worker for document processing. Store job status and output references durably so restarts and retries do not lose or duplicate the customer’s result.
See the Rivtor deployment workflow →
Pass references instead of large documents
The API should validate an upload, store it in an authorized object location and enqueue a job containing an internal identifier. Avoid embedding the whole document or a long-lived credential in the queue payload. A private object reference still needs an ownership check before a worker or browser can access it.
Consider an illustrative invoice-extraction workflow. The API creates a pending job row, stores a small test PDF and enqueues its identifier. A worker extracts text and calls an external model service. The application host runs the pipeline; it does not imply local GPU inference or a specific document-parser package is supported.
Make status transitions observable
| State | Meaning |
|---|---|
| Pending | Durable request exists but work has not started |
| Running | Worker has claimed the task |
| Completed | Output reference is committed and accessible to its owner |
| Failed | Error class and recovery decision are recorded |
Use a unique job identity so duplicate delivery does not create multiple final records. A crash can occur after an external API call but before writing completion; decide how that uncertainty is handled. Do not promise exactly-once execution from an ordinary queue.
Bound resources and retention
Set file-size, page-count, execution-time and concurrency limits. PDF and image parsing can consume substantially more memory than a small text request. Keep temporary files bounded and delete them when work finishes; durable input and output belong in appropriate storage.
Review Rivtor worker support, data limits and the broker’s memory policy. A caching-oriented Redis policy may not meet job durability requirements.
Rehearse interruption with synthetic files
Submit one supported document, an oversized file and a malformed file. Stop a worker during processing and observe whether the job becomes retryable. Confirm the final output belongs to the submitting account, and that a failed job can be investigated without revealing document contents in public logs. Compare queue identifiers, durable status and output counts rather than relying solely on a successful worker exit.