Skip to content
JavaAgentic

Type at least two characters. Try “RAG”, “pgvector” or “tool calling”.

Distributed Transactions & Saga Patterns

Why two-phase commit fails in microservices, choreography versus orchestration sagas, compensating transactions, the transactional outbox, and idempotent consumers.

Advanced7 min readUpdated
On this page

The moment your data spans two services, @Transactional stops being able to help. A saga replaces one atomic transaction with a sequence of local ones, each with an explicit undo — trading atomicity for availability, deliberately.

Key Takeaways

  • Two-phase commit trades availability for consistency in exactly the wrong direction for microservices.
  • A saga is local transactions plus compensating actions. There is no rollback, only forward correction.
  • Choreography for short flows; orchestration once the process has branches or many steps.
  • The outbox pattern is what makes "update the database and publish an event" atomic.
  • Every consumer must be idempotent, because at-least-once delivery is the only kind you get.

Why 2PC does not fit

The 2PC failure mode: participants block holding locks while the coordinator is unavailable.

Three problems make this unworkable at service scale. Locks are held across a network round trip, so throughput collapses. The coordinator is a single point of failure whose crash leaves participants blocked. And the availability of the whole transaction is the product of the availability of every participant — three services at 99.9% give you 99.7% together, and it gets worse with every addition.

Choreography

Each service reacts to events from the previous step and emits its own. No central controller.

Choreography: services subscribe to each other's events. The compensating path is the dotted line.
ChoreographedSaga.java
@Component
public class PaymentEventHandler {
 
    @KafkaListener(topics = "order-events", groupId = "payments")
    @Transactional
    public void on(OrderPlaced event) {
        // Idempotency first: at-least-once delivery means this method WILL be
        // called twice for the same event at some point.
        if (processed.exists(event.eventId())) return;
 
        try {
            Receipt receipt = gateway.charge(event.orderId(), event.total());
            outbox.publish(new PaymentCaptured(event.orderId(), receipt.id()));
        } catch (PaymentDeclinedException ex) {
            outbox.publish(new PaymentFailed(event.orderId(), ex.reason()));
        }
        processed.record(event.eventId());
    }
}

Choreography is loosely coupled and easy to extend — a new service just subscribes. Its weakness is that the business process is not written down anywhere. To answer "what happens when an order is placed?" you must read the subscriptions of six services, and cyclic dependencies are easy to create by accident.

Orchestration

A coordinator holds the state machine and tells each service what to do.

OrderSagaOrchestrator.java
@Service
public class OrderSagaOrchestrator {
 
    @Transactional
    public void handle(SagaEvent event) {
        SagaState saga = repository.lockById(event.sagaId());
 
        switch (saga.step()) {
            case STARTED -> {
                saga.advance(Step.AWAITING_PAYMENT);
                commands.send(new CapturePayment(saga.orderId(), saga.total()));
            }
            case AWAITING_PAYMENT -> {
                if (event instanceof PaymentCaptured) {
                    saga.advance(Step.AWAITING_STOCK);
                    commands.send(new ReserveStock(saga.orderId(), saga.lines()));
                } else {
                    saga.fail("payment declined");
                    // Nothing to compensate: payment was the first side effect.
                    commands.send(new CancelOrder(saga.orderId()));
                }
            }
            case AWAITING_STOCK -> {
                if (event instanceof StockReserved) {
                    saga.advance(Step.COMPLETED);
                    commands.send(new CreateShipment(saga.orderId()));
                } else {
                    // Compensate in reverse order of the forward steps.
                    saga.compensating();
                    commands.send(new RefundPayment(saga.orderId(), saga.paymentId()));
                }
            }
            case COMPENSATING -> {
                saga.fail("stock unavailable");
                commands.send(new CancelOrder(saga.orderId()));
            }
        }
        repository.save(saga);
    }
}

The whole business process is readable in one class, which is worth a great deal when you are debugging at 3am or onboarding someone. The cost is that the orchestrator knows about every participant, so it becomes a component that changes whenever the process does.

Compensating transactions

A compensation is not a rollback. The original transaction committed; the data is visible and other things may have reacted to it. A compensation is a new business action that semantically undoes the previous one.

Forward actionCompensationNote
Reserve stockRelease reservationClean
Capture paymentIssue refundVisible on the customer's statement
Send confirmation emailSend correction emailCannot be unsent
Allocate a seatRelease the seatSomeone may have seen it as taken

Two design rules follow. Compensations must be idempotent, because they will be retried. And they should be ordered last-to-first — undo the most recent step first, mirroring the forward sequence.

A third, less obvious rule: order the forward steps so the hardest-to-compensate action happens last. Sending an email cannot be undone, so send it after everything reversible has succeeded. Many saga designs become dramatically simpler with a reordering that costs nothing.

The outbox pattern

Here is the bug that every event-driven system hits: you update the database and publish an event. Those are two systems, so there is a window where one succeeds and the other does not. Publish first and the transaction may roll back, leaving an event for something that never happened. Commit first and the broker may be unreachable, leaving a change nobody hears about.

Writing the event to an outbox table in the same transaction makes state change and event publication atomic.
OutboxPublisher.java
@Component
public class OutboxPublisher {
 
    // Same transaction as the business change, so they commit or roll back together.
    @Transactional(propagation = Propagation.MANDATORY)
    public void publish(DomainEvent event) {
        outboxRepository.save(new OutboxRecord(
                UUID.randomUUID(),
                event.aggregateId(),          // becomes the partition key: ordering per aggregate
                event.getClass().getSimpleName(),
                serializer.toJson(event),
                Instant.now()));
    }
}
 
@Component
public class OutboxRelay {
 
    @Scheduled(fixedDelay = 500)
    @SchedulerLock(name = "outboxRelay", lockAtMostFor = "PT1M")
    public void relay() {
        List<OutboxRecord> batch = outboxRepository.findUnsent(PageRequest.ofSize(200));
        for (OutboxRecord record : batch) {
            kafkaTemplate.send(topicFor(record), record.aggregateId(), record.payload())
                         .whenComplete((result, ex) -> {
                             // Only mark sent on confirmation. A crash before
                             // this point means redelivery, which is fine —
                             // consumers are idempotent.
                             if (ex == null) outboxRepository.markSent(record.id());
                         });
        }
    }
}

Change Data Capture with Debezium is the alternative: it tails the database write-ahead log and publishes changes with no polling and no relay to operate. It removes the latency of the poll interval at the cost of running Debezium and Kafka Connect.

Idempotent consumers

The outbox guarantees at-least-once, never exactly-once. Duplicates are normal, not exceptional, and every consumer must handle them.

IdempotentConsumer.java
@KafkaListener(topics = "order-events")
@Transactional
public void consume(ConsumerRecord<String, String> record) {
    String eventId = header(record, "eventId");
 
    try {
        // A unique constraint on event_id makes the check atomic. A SELECT
        // followed by an INSERT has a race window; this does not.
        processedRepository.save(new ProcessedEvent(eventId, Instant.now()));
    } catch (DataIntegrityViolationException duplicate) {
        log.debug("event {} already processed, skipping", eventId);
        return;
    }
 
    handler.handle(deserialize(record.value()));
}

The unique constraint is doing the real work. A SELECT then INSERT looks equivalent and has a race window that two concurrent consumers will find.

Where the operation is naturally idempotent — setting a status to a fixed value, an upsert keyed by a business identifier — you may not need the table at all. Prefer that when you can design for it: the cheapest deduplication is an operation that does not care how often it runs.

Observability for sagas

A saga that stalls halfway is the failure mode that hurts, because nothing is obviously broken — there is simply an order that never ships. Instrument for it explicitly: a gauge of sagas per state, an alert on any saga in a non-terminal state for longer than its expected duration, and a timer for end-to-end completion. Correlate every message with the saga id so one query returns the whole flow.

What to take away

Give up on distributed atomicity and design for eventual consistency with explicit compensations. Choose choreography for short flows and orchestration for complex ones. Use the outbox so state changes and events commit together, make every consumer idempotent with a unique constraint, and alert on sagas that stop moving.

Frequently Asked Questions

Why not just use XA transactions across services?
XA holds locks on every participant for the duration of the prepare phase, which couples the availability of all of them together and destroys throughput. The coordinator is a single point of failure whose crash leaves resources locked. And most modern infrastructure — Kafka, most HTTP APIs, most cloud databases — has no XA support at all.
Choreography or orchestration?
Choreography for two or three steps where the flow is obvious. Orchestration once there are four or more, or when the sequence has branches, because with choreography the business process exists only implicitly in the event subscriptions and nobody can read it. The cost of orchestration is a component that knows about everyone.
What if a compensating transaction fails?
Retry it with backoff, because compensations must be idempotent and eventually succeed. If it still fails, the saga goes to a dead-letter state with an alert and a human resolves it. There is no clean automatic answer, which is why compensations should be designed to be as simple and as likely to succeed as possible.

Related tutorials