Practice
What is Agent Evals?
Also known as: agent evaluation, evals
Agent evals are repeatable tests that measure whether an AI agent actually completes tasks correctly — scoring real runs against expected outcomes rather than judging output by eye.
Unit tests check that a function returns the right value. Agent evals check something fuzzier: given a task, did the agent reach the right end state? A suite is a set of fixed scenarios — find the invoice from March and draft a reply — plus a way to score the result: exact checks on the final state, assertions on which tools were called, or a stronger model grading the output against a rubric. You run them on every prompt or model change.
Evals matter more as autonomy goes up, because the failure you care about is behavioural, not syntactic — the agent picked a plausible wrong tool, or stopped one step early. Velaris agents are multi-step and route across models, and both of those are things you can only tune with measurement: a cheaper model is only cheaper if it still passes. Tiered tools help here too — a read-tier step that fails is a bad answer, but a destructive-tier step that fails is why approvals exist.
Related terms
See it in action
Velaris puts agent evals to work inside a real AI Operating System.
Get early access