Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

LLM Evaluation & Benchmarks

How to evaluate LLMs and LLM applications: what public benchmarks like MMLU and HumanEval measure, their limits, and building a custom evaluation suite that reflects your real task.

Advanced4 min readUpdated
On this page

You cannot improve what you cannot measure, and LLM evaluation is genuinely hard because outputs are open-ended and non-deterministic. This tutorial covers what public benchmarks do and do not tell you, and how to build an evaluation suite that reflects your actual task — the only evaluation that should drive decisions.

Key Takeaways

  • Public benchmarks measure narrow capabilities on fixed datasets — a rough filter, not a decision.
  • Build a custom eval suite of your real inputs with known-good outcomes.
  • Evaluate open-ended output with property checks and LLM-as-judge, validated against human labels.
  • The judge is one signal, biased and imperfect — never the sole gate.

What public benchmarks measure

BenchmarkMeasuresMethod
MMLUBroad academic knowledgeMultiple-choice across subjects
HumanEvalCode generationGenerated functions pass tests
GSM8KGrade-school maths reasoningWord problems
HELMHolistic capabilitiesMany scenarios and metrics
TruthfulQAResistance to common falsehoodsQuestions that invite myths

Each captures one slice of capability on one dataset. They are genuinely useful for comparing models at a high level and tracking the field's progress.

Building a custom evaluation suite

The evaluation that matters is your task on your data. Build it deliberately:

A task-specific eval suite
record EvalCase(String input, Predicate<String> outcomeCheck, String description) {}
 
public EvalReport evaluate(ChatModel model, List<EvalCase> cases) {
    int passed = 0;
    for (EvalCase c : cases) {
        String output = run(model, c.input());
        boolean ok = c.outcomeCheck().test(output);
        if (ok) passed++;
        else log.info("FAIL: {} — {}", c.description(), output);
    }
    return new EvalReport(passed, cases.size());
}

The cases come from real usage: questions users actually ask, documents you actually process, edge cases you have actually hit. Fifty well-chosen cases that reflect your task beat any public benchmark for deciding what to ship.

Evaluating deterministic tasks

For extraction, classification and retrieval, outcomes are checkable and evaluation is precise:

// Classification: exact match against a labelled set.
assertThat(classify(input)).isEqualTo(expectedLabel);
 
// Retrieval: is the correct passage in the top-k? (The ceiling on RAG accuracy.)
assertThat(retrieve(query)).anyMatch(d -> d.sourceId().equals(expectedSource));

These run cheaply and deterministically — put them in CI. See testing AI applications.

Evaluating open-ended output

When there is no single correct answer, evaluate against properties and criteria:

Property and judge-based evaluation
// Property checks: concrete, cheap, deterministic.
assertThat(answer).containsIgnoringCase(requiredFact);
assertThat(answer.length()).isLessThan(maxLength);
 
// LLM-as-judge: for qualities you cannot assert directly.
boolean faithful = judge.evaluate("""
        Does the ANSWER follow only from the CONTEXT? Reply YES or NO.
        CONTEXT: {context}
        ANSWER: {answer}
        """, context, answer);

Regression evaluation

The most valuable ongoing use of evaluation is catching regressions. Run your suite on every prompt change, model upgrade or retrieval tweak:

// A drop in the pass rate is a regression, even if individual outputs still
// "look fine". Without this baseline, "it seems better" is a feeling.
double before = evaluate(currentConfig, suite).passRate();
double after = evaluate(candidateConfig, suite).passRate();
if (after < before) log.warn("regression: {} → {}", before, after);

This is how you upgrade a model version safely, tune a prompt without silent breakage, and know your changes help. See LLMOps for generative AI.

What good evaluation looks like

  • Reflects your task — real inputs, real outcomes.
  • Layered — deterministic checks in CI, judge-based and human review before releases.
  • Tracked over time — a pass-rate trend, not a one-off number.
  • Honest about the judge — validated against humans, used as one signal.
  • Actionable — failures point at what to fix, not just a score.

Next

Frequently Asked Questions

What do LLM benchmarks like MMLU and HumanEval measure?
MMLU measures broad knowledge across academic subjects through multiple-choice questions; HumanEval measures code generation by whether generated functions pass tests. Each captures one narrow capability on one dataset. They are useful for comparing models at a high level, but they do not tell you how a model will perform on your specific task, which may look nothing like the benchmark.
Why should I build a custom evaluation suite?
Because public benchmarks measure general capabilities that may not match your task, and models can be tuned to score well on popular ones. A custom suite of your real inputs with known-good outcomes measures what actually matters — how the model performs on the work you will give it. It is the only evaluation that should drive your decisions.
How do I evaluate outputs that have no single correct answer?
Use property-based checks (does it contain the key fact, match the schema, stay within scope) and LLM-as-judge for qualities like faithfulness or relevance, validated against some human-labelled examples. For open-ended tasks you evaluate against criteria and rubrics rather than exact matches, accepting that the evaluation is itself approximate.
Can I trust an LLM to evaluate another LLM?
As one signal, yes; as the only one, no. LLM-as-judge scales evaluation to qualities you cannot assert directly, but it has its own biases and error rate — it can favour longer answers or its own style. Validate the judge against human labels, use it alongside deterministic checks, and be sceptical of small score differences.

Related tutorials