AI and agents · By Rivtor · Updated
Stream LLM responses from FastAPI in production
A streaming FastAPI endpoint needs incremental output from the upstream service, an appropriate response format, a proxy path that does not buffer the result and client cancellation. Test time to first byte and gaps between chunks at the public endpoint, not only on localhost.
See the Rivtor deployment workflow →
Separate generation from delivery
A model client producing incremental events is only one part of the path. FastAPI, an application server, a reverse proxy and the browser must all preserve the intended delivery behaviour. A response that arrives all at once can be caused by buffering even when the application yields regularly.
This synthetic endpoint demonstrates timed server-sent events without calling a model provider. It is an illustrative diagnostic app, not a measured Rivtor streaming result:
import asyncio
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
async def events():
for index in range(3):
yield f'data: {index}\n\n'
await asyncio.sleep(0.2)
@app.get('/events')
async def stream():
return StreamingResponse(events(), media_type='text/event-stream')
Use a client that reads incrementally. A helper that waits for the complete response body cannot show when individual chunks arrived.
Test each network boundary
Compare direct local access with the public HTTPS path. Record first-event time, subsequent event times and completion. Inspect compression or buffering settings only where you control them. Set bounded upstream and application timeouts so disconnected or stalled requests do not occupy resources indefinitely.
Before using this pattern on any host, verify current proxy support and request-duration limits. The existence of a container runtime does not prove every streaming workload is supported end to end.
Add authentication and cancellation
Authorize the conversation before starting output. Decide how the client sends credentials and whether cross-origin requests are needed. Stop unnecessary upstream work when the client disconnects, and close the model response stream in cleanup. Handle partial output explicitly because an error after headers are sent cannot become a normal JSON error response.
Keep persistence outside the open stream transaction
Store request state with short database transactions. Commit a completed or partial reply according to an explicit application policy. Do not keep a PostgreSQL connection checked out for the whole generation time unless the design requires it and the capacity budget supports it. Review FastAPI deployment separately from this transport test.