Streaming LLM responses: the parts that break in production

Streaming LLM responses looks like a solved problem: open an SSE connection, forward tokens, done. That version works on your laptop and fails in production in four specific ways — dropped connections, buffering proxies, backpressure, and tool calls that break the mental model entirely.

Why stream at all

Not aesthetics. A model producing 600 tokens might take fifteen seconds to finish, and fifteen seconds of a spinner reads as broken. Streaming moves perceived latency from total time to first-token time, which is often under a second.

That’s the whole justification, and it’s enough. But it converts a simple request/response into a long-lived stateful connection, and everything below is the cost of that conversion.

SSE beats WebSockets here

Server-sent events are the right default for this shape. The data flows one way, SSE is plain HTTP so it traverses proxies and CDNs that mangle WebSocket upgrades, and browsers reconnect automatically.

WebSockets are justified when you need genuine bidirectional messaging mid-generation — interrupting, or streaming user input while the model is producing. For “the model talks, the client listens,” SSE is less machinery.

The gotcha: SSE over HTTP/1.1 shares the browser’s six-connections-per-origin limit. Open a stream in several tabs and the seventh request to your domain hangs. HTTP/2 fixes it; worth confirming your stack actually serves it.

The four things that break

Buffering proxies. Some reverse proxies and CDNs buffer responses by default, which defeats streaming completely — the client gets everything at once, at the end. The fix is X-Accel-Buffering: no plus Cache-Control: no-cache, and disabling compression on the stream. It usually works locally and fails only once deployed behind the real edge, which is the worst possible time to discover it.

Idle timeouts. Load balancers close connections with no traffic, often at 30 or 60 seconds. A model that thinks for a while before its first token trips this. Send a heartbeat comment (: ping\n\n) every 15 seconds — SSE ignores comment lines, so it costs nothing and keeps the connection alive.

Dropped connections. Mobile networks drop. Users close laptops. The browser reconnects automatically, and if you do nothing, generation restarts — you pay twice and the user sees the answer rewritten from the start.

No backpressure. A slow client can’t tell you to slow down. If you’re forwarding provider tokens as fast as they arrive into a buffer nobody drains, memory grows. Watch for it under load, not in testing.

Resumability is the real feature

The fix for dropped connections is to stop treating the stream as ephemeral.

Assign each generation an id. Write tokens to a durable buffer (Redis, or a table) as they arrive from the provider, and forward them. Number each chunk. When the client reconnects it sends Last-Event-ID, and you replay from there rather than regenerating.

This decouples two things that shouldn’t be coupled: the model’s generation and any particular client’s connection. Generation continues if the client vanishes. A second device can attach to the same run. A refresh resumes rather than restarts.

It also makes a nicer product possible — the user closes the tab, comes back, and the agent’s work continued. That’s the behaviour people expect from an OS and don’t expect from a chat box.

Tool calls break the streaming model

Here’s the part specific to agents, and it’s where naive implementations get confusing.

An agent’s output isn’t a single stream of prose. It’s prose, then a decision to call a tool, then a pause while that tool runs, then more prose informed by the result. The pause can be seconds. Structured tool-call arguments also arrive as partial JSON fragments that are meaningless until complete.

So don’t stream a token stream — stream typed events:

event: text        data: {"delta": "I'll check your calendar"}
event: tool_start  data: {"name": "calendar.list", "id": "c1"}
event: tool_end    data: {"id": "c1", "summary": "3 events found"}
event: approval    data: {"action": "Send invite to 4 people?"}
event: text        data: {"delta": "You're free Thursday"}
event: done        data: {"runId": "r_123"}

Now the client can render each phase honestly: text as it arrives, a labelled spinner during a tool call, an approval prompt that blocks. Trying to express all that as one prose stream produces a UI that either lies about what’s happening or freezes with no explanation.

Never stream partial tool-call arguments to the client as text. Buffer until the call is complete, then emit tool_start. Half-formed JSON on screen is noise.

Errors mid-stream

Once you’ve sent a 200 and started streaming, you can’t change the status code. An error at token 400 has to be an event in the stream, not an HTTP status.

Emit an explicit event: error with something actionable, and make sure the client handles it — otherwise a failed generation looks identical to one that simply stopped. Also distinguish aborted by user from failed, because they need different UI and different retry behaviour.

Always terminate deliberately with a done event. A stream that just stops is ambiguous, and the client can’t tell success from a silent drop.

The checklist

1. X-Accel-Buffering: no, no compression on the stream.
2. Heartbeat every ~15s to survive idle timeouts.
3. Durable buffer + chunk ids, resume on Last-Event-ID.
4. Typed events, not raw tokens.
5. Buffer tool-call arguments until complete.
6. Explicit error and done events.
7. Generation survives client disconnect.

Items 1 and 2 are the ones that pass locally and fail in production.

See also: what to keep in context across a long run.