Apache Kafka Architecture & Core Concepts

Kafka is not a message queue. It is a distributed, partitioned, replicated commit log, and almost every surprising behaviour follows from that one design decision.

The log#

Three consequences fall out of this picture.

Ordering is per partition. Records with the same key land in the same partition and are processed in order. Records with different keys have no ordering relationship at all. This is why choosing a key is a design decision, not an afterthought — key by orderId and each order's events stay ordered; key by nothing and Kafka round-robins with no ordering guarantee whatsoever.

Parallelism is capped by partitions. A group with more consumers than partitions leaves the extras idle. Six partitions means at most six useful consumers in one group.

Groups are independent. Billing and analytics each read every record and track their own offsets. Adding a consumer group costs the producer nothing and requires no change to it.

Replication and durability#

Each partition has one leader and some followers. All reads and writes go through the leader; followers replicate. The set of replicas caught up with the leader is the in-sync replica set.

acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
retries=2147483647
compression.type=zstd
linger.ms=10
batch.size=65536

replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

These settings only work as a set, and the failure mode when they do not is silent data loss.

acks=all means the leader waits for all in-sync replicas. But if replication factor is 3 and two followers fall behind, the ISR shrinks to just the leader — and acks=all now means acks=1. Losing that broker loses acknowledged data. min.insync.replicas=2 closes the hole by refusing writes when fewer than two replicas are in sync: you get an error instead of quiet loss.

unclean.leader.election.enable=false prevents an out-of-sync replica becoming leader, which would silently truncate records that were already acknowledged.

enable.idempotence=true gives the producer a sequence number per partition so a retry after a network blip does not append a duplicate. It costs nothing and should be on by default.

Producing#

@Component
public class OrderProducer {
 
    private final KafkaTemplate<String, OrderEvent> template;
 
    public CompletableFuture<SendResult<String, OrderEvent>> publish(OrderEvent event) {
        // The key is the ordering domain. Every event for one order lands in
        // the same partition and is therefore processed in sequence.
        var record = new ProducerRecord<>("orders", event.orderId(), event);
        record.headers()
              .add("eventId", event.eventId().getBytes(UTF_8))
              .add("eventType", event.getClass().getSimpleName().getBytes(UTF_8))
              .add("correlationId", Objects.toString(MDC.get("correlationId"), "").getBytes(UTF_8));
 
        return template.send(record).whenComplete((result, ex) -> {
            if (ex != null) {
                log.error("failed to publish {} for {}", event.eventType(), event.orderId(), ex);
            } else {
                log.debug("published to {}-{} offset {}",
                        result.getRecordMetadata().topic(),
                        result.getRecordMetadata().partition(),
                        result.getRecordMetadata().offset());
            }
        });
    }
}

linger.ms is the setting with the best return. It tells the producer to wait a few milliseconds to accumulate a batch. Ten milliseconds of added latency typically multiplies throughput several times over, because compression works far better on a batch than on a single record.

Consuming#

Consumers pull. A consumer polls for records, processes them, and commits an offset recording how far it has read.

group.id=billing
enable.auto.commit=false
auto.offset.reset=earliest
max.poll.records=500
max.poll.interval.ms=300000
session.timeout.ms=45000
heartbeat.interval.ms=3000

enable.auto.commit=false matters more than any other consumer setting. With auto-commit, offsets advance on a timer regardless of whether processing succeeded — a crash mid-batch means those records are never reprocessed and are silently lost. Commit manually after processing and you get at-least-once, which combined with an idempotent consumer gives exactly-once processing.

auto.offset.reset decides what a brand-new group does: earliest replays the entire retained history, latest starts from now. This surprises people during deploys — a typo in group.id creates a new group, and with earliest it reprocesses everything.

Rebalancing#

This is the most common Kafka production problem, and the diagnosis is nearly always the same: processing a batch takes longer than max.poll.interval.ms. The broker cannot distinguish a slow consumer from a crashed one, so it reassigns the partitions — and the recovering consumer rejoins, triggering another rebalance. Under load this becomes a loop where the group spends more time rebalancing than working.

Three fixes, in order of preference: reduce max.poll.records so each batch is smaller; make processing faster or move slow work to a thread pool while continuing to poll; or raise max.poll.interval.ms if the work genuinely takes that long. Also prefer the cooperative sticky assignor, which reassigns only the partitions that need to move instead of revoking everything.

Retention and compaction#

Kafka retains records by time or size, independent of whether anyone consumed them — retention.ms=604800000 keeps a week. That is what makes replay possible: a new consumer group can read the entire retained history, which is invaluable for backfilling a new service or recovering from a bug that corrupted a downstream store.

Log compaction is a different policy. With cleanup.policy=compact, Kafka keeps the latest record per key forever and deletes superseded ones. The topic becomes a snapshot of current state rather than a history of changes, which is exactly what a changelog or a lookup table wants. A null value is a tombstone marking the key deleted.

What to take away#

Kafka is a log, and the key you choose determines both ordering and parallelism. Set acks=all with min.insync.replicas=2 and idempotence on, or your durability guarantee is weaker than you think. Turn off auto-commit. And when throughput mysteriously collapses, look at rebalance frequency first.