Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Spring AI Observability & Monitoring

Instrument Spring AI with Micrometer and OpenTelemetry: token and cost metrics per feature, latency tracking, tracing model calls, and dashboards that catch a cost problem before the invoice does.

Advanced4 min readUpdated
On this page

You cannot operate what you cannot see, and AI features fail in ways ordinary metrics miss — a correct-looking answer that cost ten times too many tokens, a slow retrieval hidden inside an acceptable total latency. This tutorial instruments Spring AI so cost, latency and quality are visible before they become incidents.

Key Takeaways

  • Record token usage as a metric tagged by feature from day one — it is your only early warning for a cost problem.
  • Trace the full request: retrieval, generation and tools as separate spans.
  • Log metadata, not content — prompts contain personal data.
  • Turn tokens into a cost metric and alert on the daily rate.

Enable Spring AI observability

Spring AI integrates with Micrometer. Add the Actuator and a registry, and it emits observations for model calls automatically:

pom.xml
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
application.yml
management:
  endpoints:
    web:
      exposure:
        include: health, prometheus, metrics
  metrics:
    tags:
      application: ai-service
spring:
  ai:
    chat:
      observations:
        # Off by default, and rightly so — prompts may contain personal data.
        log-prompt: false

Token metrics per feature — the one that matters most

Aggregate token counts tell you nothing actionable. Tokens tagged by feature tell you which feature is expensive:

TokenMetricsAdvisor.java
public class TokenMetricsAdvisor implements CallAdvisor {
 
    private final MeterRegistry registry;
    private final String feature;
 
    public TokenMetricsAdvisor(MeterRegistry registry, String feature) {
        this.registry = registry;
        this.feature = feature;
    }
 
    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
        var sample = io.micrometer.core.instrument.Timer.start(registry);
        ChatClientResponse response = chain.nextCall(request);
        sample.stop(registry.timer("ai.latency", "feature", feature));
 
        var chatResponse = response.chatResponse();
        if (chatResponse != null && chatResponse.getMetadata().getUsage() != null) {
            var usage = chatResponse.getMetadata().getUsage();
            registry.counter("ai.tokens", "feature", feature, "type", "prompt")
                    .increment(usage.getPromptTokens());
            registry.counter("ai.tokens", "feature", feature, "type", "completion")
                    .increment(usage.getCompletionTokens());
        }
        return response;
    }
 
    @Override public String getName() { return "token-metrics"; }
    @Override public int getOrder() { return 100; }
}

Attach it per feature so the tags mean something:

ChatClient supportClient = builder
        .defaultAdvisors(new TokenMetricsAdvisor(registry, "support"))
        .build();

Turning tokens into cost

Tokens are the input; money is what people care about. Convert:

CostCalculator.java
@Component
public class CostCalculator {
 
    // Prices per 1M tokens. Keep these in configuration and update them when
    // the provider changes pricing.
    private record Pricing(double promptPerM, double completionPerM) {}
    private final Map<String, Pricing> pricing = Map.of(
            "gpt-4o-mini", new Pricing(0.15, 0.60),
            "gpt-4o", new Pricing(2.50, 10.00));
 
    public double costUsd(String model, long promptTokens, long completionTokens) {
        Pricing p = pricing.getOrDefault(model, new Pricing(0, 0));
        return (promptTokens / 1_000_000.0) * p.promptPerM()
                + (completionTokens / 1_000_000.0) * p.completionPerM();
    }
}

Record it as a metric and you can alert on the daily rate.

Tracing the full request

A total latency of 3.5 seconds is not actionable. A trace showing retrieval took 40ms and generation took 3.4s tells you exactly where to look.

Tracing spans
@Observed(name = "rag.request")
public RagAnswer answer(String question) {
    List<Document> context = observation("rag.retrieve",
            () -> retriever.retrieve(question));   // span 1
 
    return observation("rag.generate",
            () -> generate(question, context));    // span 2
}

With OpenTelemetry exporting to a backend, you see the waterfall and can attribute latency correctly. Add the OTLP exporter:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

The dashboard that earns its keep

A minimal but sufficient dashboard tracks, per feature:

  • Cost per hour (with an alert threshold) — catches runaway loops and context bloat.
  • P95 latency, split into retrieval and generation — tells you what to optimise.
  • Requests per minute — capacity planning and anomaly detection.
  • Error rate by type — provider errors vs your errors.
  • Tokens per request trend — a rising trend often means context creeping upward.

Logging: metadata by default, content never (casually)

// Good: safe to log, genuinely useful.
log.info("model call feature={} model={} promptTokens={} latencyMs={}",
        feature, model, usage.getPromptTokens(), latency);
 
// Dangerous: the prompt may contain personal data, and logs go to a much
// wider audience than the model.
// log.info("prompt: {}", fullPrompt);   // don't

Next

Frequently Asked Questions

How do I track token usage in Spring AI?
Read the usage metadata from the ChatResponse — getMetadata().getUsage() gives prompt and completion tokens — and record it as a Micrometer counter tagged by feature. Spring AI also emits observations automatically when observability is enabled, which you can export to Prometheus or an OpenTelemetry backend.
How do I monitor LLM costs before the bill arrives?
Convert token counters into a cost metric using each model's per-token price, tag it by feature and model, and alert when the daily rate exceeds a threshold. Cost problems are almost always a runaway loop or an oversized context sent on every request — a per-feature cost dashboard makes the culprit obvious.
Should I log prompt and response content?
Be very careful. Prompts and responses routinely contain personal data, and a log aggregator is a far wider audience than the model. Log metadata — lengths, token counts, latencies, model, feature — by default, and log content only in a controlled, access-restricted way with a clear retention policy, if at all.
What should I trace in an AI application?
The full path of a request: retrieval (how long, how many chunks, what scores), each model call (latency, tokens, model), and any tool invocations. A distributed trace that shows retrieval took 40ms and generation took 3s tells you immediately where to optimise, which aggregate metrics cannot.

Related tutorials