Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Agent Evaluation & Testing

How to evaluate and test AI agents: trajectory analysis, benchmarking, hallucination detection, outcome verification and human-in-the-loop evaluation — with Java patterns.

Advanced4 min readUpdated
On this page

Agents are harder to test than single model calls because the path changes every run and the failure modes are subtler — an agent can reach a wrong answer through any of a dozen steps, or report a success that never happened. This tutorial covers evaluating agents on outcomes and trajectories rather than exact behaviour.

Key Takeaways

  • Test deterministic parts (tools, guardrails, parsing) normally; evaluate behaviour on a fixed task set.
  • Assert on outcomes and properties, not exact trajectories — the path varies.
  • Trajectory logging is mandatory; it is the only way to debug a wrong conclusion.
  • The worst hallucination is a reported success that did not happen — verify outcomes independently.

Two layers, again

As with testing AI applications, split the problem. The tools, guardrails and parsing are deterministic — mock the model and test them on every commit. The agent's behaviour is non-deterministic — evaluate it on a fixed task set, sparingly.

Trajectory logging is not optional

You cannot debug an agent from its final answer. Record every step:

Trajectory record
record Step(int index, String thought, String tool, String args, String observation, long ms) {}
 
class TrajectoryRecorder {
    void record(String runId, Step step) {
        repository.append(runId, step);
        log.info("run={} step={} tool={} ms={}", runId, step.index(), step.tool(), step.ms());
    }
}

When an agent reaches a wrong conclusion, you replay the trajectory and see exactly which tool returned misleading data or where the reasoning turned. Without it, agent debugging is guesswork.

Evaluating on outcomes

Because the path varies, assert on the outcome and on safety, not on a specific sequence:

Outcome evaluation
@ParameterizedTest
@MethodSource("evaluationTasks")
void agentReachesCorrectOutcomeSafely(EvalTask task) {
    AgentRun run = agent.execute(task.input());
 
    // 1. Did it reach a correct outcome? (Property, not exact answer.)
    assertThat(run.result()).satisfies(task.outcomeCheck());
 
    // 2. Did it stay within budget? A correct answer that took 40 steps is a
    //    problem even if the answer is right.
    assertThat(run.steps()).isLessThanOrEqualTo(task.maxSteps());
 
    // 3. Did it take only safe actions? No write tool without approval.
    assertThat(run.actions()).allMatch(this::wasAuthorised);
}

Hallucination and outcome verification

The dangerous failure is not a visibly wrong answer — it is a confident, plausible answer that is false, or a reported success that did not occur. Verify against ground truth:

Verifying claims and actions
public VerificationResult verify(AgentRun run) {
    List<Issue> issues = new ArrayList<>();
 
    // Do cited sources exist and say what the agent claims?
    for (Citation c : run.citations()) {
        if (!sourceExists(c) || !sourceSupports(c)) {
            issues.add(Issue.unsupportedClaim(c));
        }
    }
    // Did reported actions actually happen? The agent saying "I updated the
    // record" is not evidence the record was updated.
    for (ReportedAction a : run.reportedActions()) {
        if (!actuallyHappened(a)) {
            issues.add(Issue.phantomAction(a));
        }
    }
    return new VerificationResult(issues);
}

Benchmarking and regression

Maintain a fixed set of tasks with known-good outcomes and run it on every meaningful change:

// Track over time. A drop in outcome success or a rise in safety violations is
// a regression, even if individual runs still "look fine".
double successRate = evalTasks.stream().filter(this::passes).count() / (double) evalTasks.size();
double safetyViolationRate = evalTasks.stream().filter(this::hadUnsafeAction).count()
        / (double) evalTasks.size();

Without this, "the agent seems better after my prompt change" is a feeling, not a measurement — and agents regress in non-obvious ways.

LLM-as-judge for agent quality

For qualities you cannot assert directly — was the reasoning sound, was the answer complete — use a model as judge, accepting its imperfection:

boolean sound = judge.evaluate("""
        Given the evidence gathered, is the agent's conclusion justified?
        Answer YES or NO with a one-line reason.
        """, run.trajectory());

Use it alongside outcome checks and trajectory review, never as the only gate. See LLM evaluation & benchmarks.

Human-in-the-loop evaluation

For high-stakes agents, sample runs for human review — especially early, and especially the runs the automated checks flagged as borderline. Humans catch the subtle wrongness that automated evaluation misses, and their labels improve your judge and your task set. See human-in-the-loop systems.

Next

Frequently Asked Questions

How do you test a non-deterministic agent?
Test the deterministic parts normally with mocks — tool implementations, parsing, guardrails. For the agent's behaviour, evaluate on a fixed set of tasks with known-good outcomes, asserting on properties and outcomes rather than exact trajectories. Because the path varies, you check whether the agent reached a correct result and took safe actions, not whether it followed one specific route.
What is trajectory analysis?
Examining the full sequence of an agent's steps — every prompt, tool call and observation — rather than just its final answer. It is essential for debugging agents, because a wrong conclusion could come from any step, and only the trajectory shows which tool returned misleading data or where the reasoning went wrong.
How do you detect agent hallucinations?
Verify claims against ground truth where you can — check that cited sources exist and say what the agent claims, confirm that reported actions actually happened, and use a separate model to judge whether the answer follows from the evidence. The most dangerous hallucination is a reported success that did not occur, so verify outcomes independently rather than trusting the agent's own summary.
Should agent evaluations run in CI?
Run deterministic tests (tools, guardrails, parsing) on every commit. Run behavioural evaluations — which call real models, cost money and are slower — on a schedule or before releases, guarded to skip without an API key. Track outcome success rate and safety-violation rate over time so a regression is visible.

Related tutorials