Concept
What is Latency?
Also known as: response time, time-to-first-token
Latency is the delay between a request and its response — for language models, usually split into time-to-first-token and the rate tokens stream after it, and for agents, the sum across every step.
Two numbers describe model latency and they behave differently. Time-to-first-token is the wait before anything appears — it grows with how much context you sent, since the whole prompt must be processed first. Then tokens stream at some rate per second, so a long answer takes longer regardless. Streaming exists because perceived latency is not measured latency: text appearing in 300ms feels responsive even if the full answer takes ten seconds. Bigger models are slower, longer prompts are slower, and both are choices you make.
Agent latency is a different animal, because the steps are sequential. A multi-step Velaris run pays for each inference, plus the real-world time of every tool call — an API that takes two seconds, a rate limit that forces a backoff, a mesh lookup. Then there's the honest part: an approval gate on a destructive action means the run waits for a person, and that's hours, not milliseconds. It's a deliberate trade. Routing cheap steps to fast models buys back seconds; not sending the wrong email is worth more.