Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

AI Observability & LLM Tracing

Observe LLM applications in production: distributed tracing of model and retrieval calls, LangFuse and OpenTelemetry GenAI conventions, span attributes, and cost dashboards for Java teams.

Advanced4 min readUpdated
On this page

Observing an LLM application needs everything normal observability provides plus concerns unique to AI: cost per call, retrieval quality, and output quality that a fast, error-free system can still get wrong. This tutorial covers LLM-specific tracing with OpenTelemetry and tools like LangFuse, building on Spring AI observability.

Key Takeaways

  • LLM observability adds cost, retrieval quality and output quality to the usual signals.
  • Trace the full path as connected spans — retrieval, generation, tools.
  • OpenTelemetry GenAI conventions standardise LLM span attributes across services.
  • LangFuse and similar tools give a purpose-built LLM view; wire them alongside your traces.

What LLM observability adds

A traditional dashboard shows latency, errors and throughput. An LLM system can be green on all three while:

  • Spending ten times the necessary tokens on every call.
  • Retrieving irrelevant context and answering worse than last week.
  • Drifting in output quality after a provider updates a model.

None of these show up as an error. LLM observability is about seeing them.

Distributed tracing of the request path

The single most useful thing: connected spans showing where time and tokens go.

Traced RAG request
@Observed(name = "rag.request")
public RagAnswer answer(String question) {
    // Each step is a child span, so the trace shows the waterfall.
    List<Document> context = traced("rag.retrieve", () -> {
        var docs = retriever.retrieve(question);
        span().tag("chunks", docs.size());
        span().tag("top_score", topScore(docs));
        return docs;
    });
 
    return traced("rag.generate", () -> {
        var response = generate(question, context);
        span().tag("gen_ai.usage.prompt_tokens", response.promptTokens());
        span().tag("gen_ai.usage.completion_tokens", response.completionTokens());
        span().tag("gen_ai.request.model", response.model());
        return response;
    });
}

A total latency of 3.5s is not actionable; a trace showing retrieval at 40ms and generation at 3.4s tells you exactly where to optimise.

OpenTelemetry GenAI conventions

Rather than inventing attribute names, follow the OpenTelemetry GenAI semantic conventions so your traces are consistent and portable:

// Standard attribute names mean any OTel backend understands your LLM spans,
// and traces are consistent across every service in your system.
span.setAttribute("gen_ai.system", "openai");
span.setAttribute("gen_ai.request.model", "gpt-4o-mini");
span.setAttribute("gen_ai.request.temperature", 0.2);
span.setAttribute("gen_ai.usage.input_tokens", promptTokens);
span.setAttribute("gen_ai.usage.output_tokens", completionTokens);

Export with the OTLP exporter to any compatible backend — see Spring AI observability for the setup.

Purpose-built tools: LangFuse and friends

General observability backends show traces; LLM-specific tools add prompt management, output evaluation and agent trajectory views. LangFuse is a common open-source choice:

// Capture a trace with the model call, retrieval and outcome, so LangFuse can
// show token cost, latency, and let you review and score outputs over time.
langfuse.trace(traceId)
        .generation(g -> g.model(model).input(prompt).output(response)
                .usage(promptTokens, completionTokens))
        .score("faithfulness", judgeScore);

The cost dashboard

Cost is the signal most unique to LLM systems and the one most likely to surprise you. Track it, tagged, with alerts:

// Per-feature, per-model cost — so "which feature is expensive?" has an answer.
registry.counter("gen_ai.cost.usd", "feature", feature, "model", model)
        .increment(costCalculator.costUsd(model, promptTokens, completionTokens));

The dashboard should show cost per feature per hour with an alert threshold — this is what catches a runaway agent loop in minutes instead of on the invoice.

Output quality monitoring

The hardest and most important: is the output still good? You cannot check every response, so sample:

// Sample a fraction of production responses for evaluation — property checks,
// an LLM judge, or human review — and track the quality trend over time. A
// drift downward is a regression that latency and error metrics never show.
if (sampler.shouldSample()) {
    qualityMonitor.evaluate(request, response);
}

A provider can change a model under a version alias, or your input distribution can shift; sampled quality monitoring is how you notice. See LLM evaluation & benchmarks.

The observability checklist

  • Connected traces: retrieval, generation, tools as spans
  • OpenTelemetry GenAI conventions for portable, consistent attributes
  • Token and cost per feature, with alerting on the rate
  • Latency split into retrieval and generation
  • Sampled output-quality monitoring for drift
  • Agent trajectory logging for debugging
  • Metadata logged, content handled carefully (personal data)

Next

Frequently Asked Questions

What is the difference between LLM observability and normal observability?
LLM observability adds concerns normal observability lacks: token usage and cost per call, prompt and response tracking, retrieval quality, and the quality of outputs themselves — which are non-deterministic. You still need latency, errors and throughput, but an LLM system can be fast and error-free while producing wrong, expensive or drifting output, which only LLM-specific observability catches.
What is LangFuse?
An open-source LLM observability and tracing platform that captures traces of model calls, retrieval steps and agent trajectories, with token and cost tracking, prompt management and evaluation. It gives you a purpose-built view of LLM application behaviour that general observability tools do not, and integrates with Java through its API and framework hooks.
What are the OpenTelemetry GenAI conventions?
A set of semantic conventions that standardise how to represent LLM operations in OpenTelemetry traces — span names and attributes for the model, token counts, temperature and so on. Following them means your LLM traces work with any OpenTelemetry-compatible backend and are consistent across services, rather than each team inventing its own attributes.
What should I trace in an LLM application?
The full request path as connected spans: retrieval (duration, chunks, scores), each model call (latency, tokens, model, temperature), tool invocations, and the final outcome. Connected spans let you see that retrieval took 40ms and generation took 3s, attribute cost per step, and debug where a slow or wrong result came from — which aggregate metrics cannot.

Related tutorials