Distributed Tracing & Observability

When a request crosses four services, no single log file explains where the time went. A trace does — it reconstructs the whole call tree with timings, which turns "checkout is slow" into "the inventory lookup takes 800ms in this one branch".

The model#

A span carries a name, start and end time, a parent span id, attributes (key/value metadata), events (timestamped annotations) and a status. The SpanContext — trace id, span id, sampling flag — is what travels between services.

Zero-code instrumentation#

java -javaagent:/opt/opentelemetry-javaagent.jar \
     -Dotel.service.name=order-service \
     -Dotel.resource.attributes=deployment.environment=production,service.version=2.4.1 \
     -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
     -Dotel.traces.sampler=parentbased_traceidratio \
     -Dotel.traces.sampler.arg=0.1 \
     -jar app.jar

That single flag instruments Spring MVC, WebClient, RestTemplate, JDBC, JPA, Kafka, RabbitMQ, Redis and Elasticsearch, and propagates context across all of them. It is the highest return on effort available in observability.

For a Spring-native approach without an agent, Micrometer Tracing with the OTel bridge gives the same propagation using the Observation API:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Adding business context#

The agent knows about HTTP and SQL. It does not know that this request is a high-value order for a premium customer — and that is often exactly the dimension you need to slice by.

@Service
public class OrderService {
 
    private final ObservationRegistry registry;
 
    public Order place(PlaceOrderCommand command) {
        return Observation.createNotStarted("order.place", registry)
                // Low cardinality: safe as a metric dimension too.
                .lowCardinalityKeyValue("order.channel", command.channel())
                .lowCardinalityKeyValue("customer.tier", command.tier().name())
                // High cardinality: searchable in traces, never a metric tag.
                .highCardinalityKeyValue("order.id", command.orderId())
                .observe(() -> {
                    var order = repository.save(command.toOrder());
                    events.publish(new OrderPlaced(order.id()));
                    return order;
                });
    }
}

The low/high cardinality split matters. Micrometer promotes low-cardinality keys to metric tags and keeps high-cardinality ones on the span only. Tagging a metric with an order id would create one time series per order and take the monitoring stack down.

Propagation#

The W3C standard puts context in two headers:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             │  │                                │                │
             │  trace-id (16 bytes)              parent-span-id   flags (01 = sampled)
             version
tracestate: acme=abc123,vendor=xyz

Baggage propagates business context alongside the trace:

try (var scope = tracer.createBaggageInScope("tenant.id", tenantId)) {
    // Every downstream service can read tenant.id from baggage without it
    // being a parameter on every method signature.
    return processOrder(command);
}

Baggage is powerful and easy to misuse. It travels on every request in the trace, so a large baggage payload adds bytes to every hop — and it crosses trust boundaries, so never put anything sensitive or security-relevant in it.

Across a broker, context must be written to message headers on publish and read on consume. Auto-instrumentation does this for Kafka and RabbitMQ; a hand-rolled transport will silently break the trace, and the symptom is a trace that ends at the producer with no visible error.

Sampling#

Tracing every request at scale is expensive in network, storage and money. Two strategies:

Head sampling decides at the start of the trace, before you know whether anything interesting happened. It is cheap and it throws away errors as readily as successes.

Tail sampling buffers complete traces in the collector and decides afterwards, which lets you keep exactly what matters:

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-all-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }
  batch:
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

Keep every error, every slow trace, and 5% of normal traffic. That is a large cost reduction with almost no loss of diagnostic value, because nobody investigates a fast, successful request.

Use parentbased_traceidratio in services so the sampling decision made at the edge is honoured throughout — otherwise you get partial traces where some services sampled and others did not.

Correlating with logs#

<pattern>%d %-5level [%X{traceId:-},%X{spanId:-}] %logger{36} - %msg%n</pattern>

Micrometer Tracing populates the MDC with traceId and spanId automatically. With those in your structured logs, a slow span in Tempo links directly to the log lines it produced, and a suspicious log line links back to the full request trace. That round trip is where tracing stops being a nice diagram and starts saving time during incidents.

What to take away#

Start with the Java agent and an OTLP collector — it is a flag, not a project. Add spans only for business operations worth naming, and keep high-cardinality values off metrics. Use tail sampling to keep errors and slow traces while discarding the boring majority, and put the trace id in your logs so the three signals connect.