Production Observability — Full Stack

An observability stack is infrastructure like any other: it needs capacity planning, cost control and an owner. Built well it is the difference between a fifteen-minute incident and a three-hour one.

The pipeline#

The two-tier collector is the pattern worth adopting. A DaemonSet collector on each node handles local buffering with minimal latency; a gateway Deployment does tail sampling and enrichment, which needs to see complete traces and therefore cannot be per-node.

receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
 
processors:
  # First in the chain. Without it a traffic spike OOMs the collector and you
  # lose observability exactly when you need it.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
 
  batch:
    timeout: 5s
    send_batch_size: 8192
 
  # Adds pod, node and namespace attributes automatically.
  k8sattributes:
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.node.name, k8s.deployment.name]
 
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
 
exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  prometheusremotewrite/mimir:
    endpoint: http://mimir:9009/api/v1/push
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, tail_sampling, batch]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [prometheusremotewrite/mimir]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [loki]

memory_limiter first in every pipeline is not optional. A collector that OOMs during a traffic spike takes your visibility with it at the exact moment you need it, and the resulting incident is diagnosed blind.

Application side#

java -javaagent:/opt/opentelemetry-javaagent.jar \
     -Dotel.service.name=order-service \
     -Dotel.resource.attributes=service.version=${APP_VERSION},deployment.environment=production \
     -Dotel.exporter.otlp.endpoint=http://$(NODE_IP):4317 \
     -Dotel.traces.sampler=parentbased_always_on \
     -Dotel.metrics.exporter=otlp \
     -Dotel.logs.exporter=otlp \
     -jar app.jar

Note parentbased_always_on at the application. Sampling happens in the gateway with tail sampling, which needs complete traces to decide — sampling at the application would discard spans before the decision could be made.

Export logs through OTLP too, and they arrive with trace context already attached, which removes the correlation work entirely.

Retention and cost#

Mimir, Loki and Tempo all store on object storage, which makes long retention affordable — the expensive part is ingest and indexing, not bytes at rest.

The single biggest cost lever is cardinality for metrics and sampling for traces. One label carrying a user id can multiply metric cost by a thousand; tail sampling that keeps errors and 5% of the rest typically cuts trace cost by 90% with no diagnostic loss.

Cardinality is worth enforcing rather than hoping for. A Micrometer MeterFilter can cap the number of distinct tag values on a meter and fold the overflow into a single bucket, which turns an accidentally unbounded label into a known ceiling instead of an invoice and, eventually, an ingestion outage.

Alerts that earn their page#

groups:
  - name: slo
    rules:
      - alert: ErrorBudgetBurningFast
        # Multi-window burn rate: catches both a sudden severe outage and a
        # slow degradation, without paging for a single bad minute.
        expr: |
          (
            sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) by (application)
            / sum(rate(http_server_requests_seconds_count[5m])) by (application)
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_server_requests_seconds_count{status=~"5.."}[1h])) by (application)
            / sum(rate(http_server_requests_seconds_count[1h])) by (application)
          ) > (14.4 * 0.001)
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: '{{ $labels.application }} is burning its error budget 14x too fast'
          runbook: 'https://runbooks.acme.com/error-budget-burn'
          dashboard: 'https://grafana.acme.com/d/service-overview?var-app={{ $labels.application }}'

Multi-window burn-rate alerting is the technique that most reduces noise. A single-window alert either pages for a two-minute blip or misses a slow burn; requiring both a short and a long window to breach catches real problems and ignores transients.

Three annotations belong on every alert: what is wrong in plain language, a runbook link, and a dashboard link pre-filtered to the affected service. The person paged may have never seen this alert before.

Runbooks#

A runbook that gets used has four sections and fits on one screen: what this alert means, how to confirm it is real (a specific query or dashboard), the immediate mitigation (scale up, roll back, disable a feature flag), and how to investigate the cause once the bleeding stops.

Write them at the moment you create the alert, while the reasoning is fresh. A runbook written six months later by someone reconstructing intent is worse than none, because it carries false authority.

Review them after each incident. The most valuable output of a postmortem is usually a runbook correction, since the person on call discovered exactly where the existing one was wrong.

What to take away#

Instrument with OpenTelemetry and export through a collector, so backends are a configuration choice. Put a memory limiter first in every pipeline and tail-sample traces at the gateway. Alert on symptoms with multi-window burn rates, attach a runbook and a dashboard to every alert, and delete alerts that never require action.