The Microservices Observability Stack

Three services can be debugged by reading logs. Thirty cannot. Observability is the difference between knowing something is wrong and knowing what to do about it, and it is assembled from three signals that answer different questions.

What each signal is for#

Trying to make one signal do another's job is where teams waste effort. Logging every request to compute an error rate is expensive and slow compared to a counter. Adding a metric per user is impossible. Reconstructing a call tree from timestamps across five log files is what traces exist to avoid.

Metrics with Prometheus#

management:
  endpoints:
    web:
      exposure:
        include: 'health,info,prometheus'
  metrics:
    tags:
      application: ${spring.application.name}
      environment: ${app.environment}
  distribution:
    percentiles-histogram:
      http.server.requests: true
    # Histogram buckets let Prometheus compute accurate quantiles across
    # instances. Client-side percentiles cannot be aggregated correctly.
    slo:
      http.server.requests: 50ms,100ms,200ms,500ms,1s,2s

The histogram detail matters more than it looks. A p99 computed per instance cannot be averaged into a fleet p99 — that is not how percentiles work. Exporting buckets lets histogram_quantile compute the real figure across every instance.

PromQL for the questions you actually ask:

# Error rate as a percentage, by service
100 * sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (application)
    / sum(rate(http_server_requests_seconds_count[5m])) by (application)
 
# p99 latency by endpoint
histogram_quantile(0.99,
  sum(rate(http_server_requests_seconds_bucket[5m])) by (le, uri, application))
 
# Connection pool exhaustion — pending acquisitions is the leading indicator
hikaricp_connections_pending{application="order-service"} > 0
 
# Kafka consumer lag, the single best queue-health signal
sum(kafka_consumer_fetch_manager_records_lag_max) by (client_id, topic)
 
# Which job is producing the most series — run this when Prometheus struggles
topk(10, count by (__name__, job)({__name__=~".+"}))

That last query is the one to remember. Cardinality problems are the most common way an observability stack falls over, and they are always caused by a label carrying an unbounded value — a user id, a raw path with an id in it, a session token.

Cardinality discipline#

Spring templates the uri tag automatically, so /orders/4711 and /orders/4712 both record as /orders/{id}. Custom metrics do not get this for free — a counter tagged with a raw path will grow without limit.

The rule of thumb: a label is safe if you can enumerate its possible values on a whiteboard. High cardinality belongs on spans and log lines, where it costs storage rather than an explosion of time series.

Logs with Loki#

Loki indexes labels rather than full text, which makes it dramatically cheaper than Elasticsearch for the same volume. The trade is that you filter by label first, then grep within the result:

{application="order-service", environment="production"} |= "ERROR"
 
# Parse structured JSON and filter on a field
{application="order-service"} | json | correlationId = "b7f3a1c2-9e04-4d1a"
 
# Derive a metric from logs when no counter exists
sum(rate({application="order-service"} | json | level="ERROR" [5m])) by (logger)

The same cardinality rule applies to Loki labels — a label per correlation id would create a stream per request. Put identifiers in the log body as structured fields and let the parser filter on them.

Golden signals#

For every user-facing service, four numbers cover most of what matters:

Latency — how long a request takes. Track p50, p95 and p99 separately, and split successful from failed requests, because fast failures otherwise flatter your average.

Traffic — requests per second. Not a health signal on its own, but essential context: a drop in errors because traffic collapsed is not an improvement.

Errors — the rate of failed requests. Count 5xx as errors; count 4xx separately, since those are usually the client's problem and alerting on them pages you for someone else's bug.

Saturation — how full the service is. Thread pool utilisation, connection pool pending acquisitions, queue depth, heap after GC. This is the leading indicator; the other three tell you it already went wrong.

Alerts worth having#

groups:
  - name: service-slos
    rules:
      - alert: HighErrorRate
        expr: |
          100 * sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (application)
              / sum(rate(http_server_requests_seconds_count[5m])) by (application) > 2
        for: 5m
        labels: { severity: critical }
        annotations:
          summary: '{{ $labels.application }} error rate is {{ $value | printf "%.1f" }}%'
          runbook: 'https://runbooks.acme.com/high-error-rate'
 
      - alert: ConsumerLagGrowing
        expr: sum(kafka_consumer_fetch_manager_records_lag_max) by (client_id) > 10000
        for: 10m
        labels: { severity: warning }
 
      - alert: JobHasNotSucceeded
        # Catches the failure mode that failure-alerting misses entirely:
        # a job that stopped being scheduled at all.
        expr: time() - max(job_last_success_timestamp_seconds) by (job) > 93600
        labels: { severity: critical }

Three properties separate a useful alert from noise. A for clause, so a momentary spike does not page anyone. A runbook link, because the person woken at 3am may never have seen this alert before. And a symptom-based expression — error rate rather than CPU, because high CPU with happy users needs no action at all.

Making the signals connect#

The highest-value integration work is linking the three. Put traceId in every log line via the MDC, and configure Grafana's derived fields so a trace id in a Loki result becomes a link into Tempo. Add exemplars to your histograms so a spike on a latency graph links to an actual slow trace.

Once that is wired, an investigation goes: alert fires → dashboard shows which service and when → click an exemplar → land on a slow trace → click the span → read its log lines. That path is what makes observability feel like a tool rather than three separate products.

What to take away#

Use metrics to detect, traces to locate, logs to explain — and link them so the path between is one click. Keep labels low-cardinality and check the top series producers when the stack strains. Alert on the four golden signals with a for clause and a runbook, and add an alert for jobs that simply stop happening.