Skip to content

AI and agents · By Rivtor · Updated

Deploy a FastAPI LLM Chatbot Backend with SSE Streaming

Deploy the FastAPI chatbot as a Python API that calls your external LLM provider. Use server-sent events for incremental replies, keep the model key on the server and verify streaming with a synthetic request before connecting the real model.

See the Rivtor deployment workflow →

Draw the application and model boundary

A FastAPI chatbot can call a separately hosted model API. The application host runs request handling and application data access; it does not thereby provide model inference hardware. State this distinction when estimating hosting cost or describing required resources.

An example service accepts an authenticated message, checks ownership of the conversation, calls the selected model provider and stores the resulting message. Each of those steps can fail independently. Keep API and model credentials outside browser code and committed source.

Test FastAPI streaming with a local fixture

From a new Python virtual environment, install FastAPI and Uvicorn and record the validated dependency versions in requirements.txt. Save this fixture as main.py. It sends three synthetic chunks through server-sent events (SSE) without a model key, provider call or database.

import asyncio
import json

from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()


@app.get("/health")
def health():
    return {"ok": True}


@app.get("/demo/stream")
async def demo_stream():
    async def events():
        for chunk in ("Hello", " from", " FastAPI"):
            yield f"data: {json.dumps({'delta': chunk})}\n\n"
            await asyncio.sleep(0.4)
        yield "event: done\ndata: {}\n\n"

    return StreamingResponse(events(), media_type="text/event-stream")
python -m pip install fastapi uvicorn
python -m uvicorn main:app --host 0.0.0.0 --port 8000

In another terminal, check that /health returns {"ok":true}. The second command should print three data: messages in succession and one done event. -N disables curl's own output buffering.

curl --fail http://127.0.0.1:8000/health
curl --fail --no-buffer http://127.0.0.1:8000/demo/stream

The fixture was checked on localhost with FastAPI 0.165.0 and Uvicorn 0.54.0. Repeat the check through your deployed URL to test the hosting path.

Deploy the chatbot backend on Rivtor

To check your own app on Rivtor, create a Rivtor account and set up the CLI. From the project directory, run npx rivtor deploy --dry-run. Verify the FastAPI service root, the ASGI entry point main:app, the listener port and the health route. Use the framework configuration reference if your repository needs an override.

After you review the plan, deploy with npx rivtor deploy. Repeat both checks using the returned public API URL. An HTTP 200 confirms the request opened. To verify streaming, check that chunks arrive before the final event, including through any proxy between you and the app. If all chunks arrive at once, repeat the request directly to the API and inspect proxy buffering separately from client rendering. Keep the synthetic route out of your production app.

Add limits before adding traffic

BoundaryLimit to define
User inputLength, accepted format and authorized conversation
Model callTimeout, output budget and concurrent requests
ApplicationPer-user rate and maximum active work
PersistenceRetention, storage growth and sensitive data handling
CostApplication resources and external API usage separately

Do not retry every upstream error automatically. A timeout can occur after a provider has processed a request. Repeated calls may increase spend or create inconsistent conversation state. Record whether a request is pending, completed or failed.

Choose synchronous, streaming or queued work

A short reply may use a normal HTTP response. Incremental output needs a supported streaming path and client cancellation. A long document-processing task usually needs a queue and a job status route. A background task living only inside a request process is not durable across restarts.

For Rivtor, follow FastAPI deployment and review the selected plan’s memory and idle behaviour. An external model key is an application secret; provisioning PostgreSQL does not configure that provider account.

Compare hosting for a FastAPI LLM app

Use the same health and streaming checks on each candidate. Review Render's FastAPI setup and separate background workers, Railway's FastAPI deploy guide, or Vercel's FastAPI support and function limits. Compare request duration, response buffering, active connection limits, database locality and the cost of any separate queue or worker. For a larger workload, see the FastAPI hosting comparison.

Verify failures as well as the happy path

Use synthetic conversations. Exercise a normal reply, invalid input, unauthorized conversation access, an upstream timeout and an application restart. Confirm one user cannot read another user’s messages. Inspect logs for accidental prompt or token exposure. Record the tested model API and client version; a completed site build is not evidence that any live model request was executed.

References