Kafka in Production

Kafka is reliable when configured correctly and quietly lossy when not. Most production incidents trace back to a handful of settings and a handful of metrics nobody was watching.

Sizing#

Partition count follows from throughput and parallelism:

partitions = max(
    target_throughput / single_partition_throughput,
    target_throughput / single_consumer_throughput
)

A partition handles roughly 10MB/s of writes on typical hardware; a consumer handles whatever your processing allows, which is usually the tighter constraint. For 50MB/s with a consumer that manages 5MB/s, you need 10 partitions for the write side and 10 for the read side — so 12 to 15 with headroom.

Add headroom deliberately, because adding partitions later has a real cost: it changes hash(key) % partitions, so existing keys can move to a different partition and their ordering relative to past events breaks. Doubling partitions on a busy topic is not a routine operation.

At the cluster level, total partitions across all topics and replicas is the number that stresses a broker. Each one costs file handles and memory, and leader election after a broker loss is proportional to how many that broker led. A few thousand per broker is comfortable; tens of thousands makes recovery slow enough to become the incident.

Retention and compaction#

kafka-configs --bootstrap-server kafka:9092 --entity-type topics --entity-name orders \
  --alter --add-config retention.ms=604800000,segment.bytes=1073741824,min.insync.replicas=2

Retention is independent of consumption — Kafka does not delete a record because someone read it, and it does delete one nobody read once the window passes. That asymmetry is what enables replay and what makes a slow consumer a data-loss risk: if lag exceeds retention, records expire before being processed and are gone silently.

Compaction keeps a tombstone (a null value) to mark deletion, and delete.retention.ms controls how long tombstones survive. Set it longer than your slowest consumer's downtime, or a consumer that was offline will never learn a key was deleted.

Metrics that predict incidents#

# Durability at risk
kafka_server_replicamanager_underreplicatedpartitions > 0
 
# Controller problem
sum(kafka_controller_kafkacontroller_activecontrollercount) != 1
 
# Consumer lag with a rate-of-change check: a large but shrinking lag after
# a deploy is normal; a growing one is not.
kafka_consumergroup_lag > 100000
  and deriv(kafka_consumergroup_lag[15m]) > 0

Consumer lag deserves that second clause. Raw lag alerts fire constantly after any restart or backfill and get ignored. Alerting on lag that is both large and increasing catches the real problem — a consumer that cannot keep up — without paging for normal catch-up.

The CLI worth knowing#

# What does this topic look like, and are all replicas healthy?
kafka-topics --bootstrap-server kafka:9092 --describe --topic orders
 
# Any partition whose ISR is smaller than its replica set
kafka-topics --bootstrap-server kafka:9092 --describe --under-replicated-partitions
 
# Consumer group state and lag per partition — the first command in an incident
kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group billing
 
# Reset a group to reprocess. Always --dry-run first, and the group must be idle.
kafka-consumer-groups --bootstrap-server kafka:9092 --group billing \
  --topic orders --reset-offsets --to-datetime 2026-07-26T00:00:00.000 --dry-run
 
# Disk usage per broker and log directory
kafka-log-dirs --bootstrap-server kafka:9092 --describe --json | jq

Offset reset is the operation that recovers from a bad deploy that processed messages incorrectly: stop the consumers, reset to a timestamp before the bad code shipped, and let them reprocess. It only works because consumers are idempotent — which is the practical payoff for insisting on that.

Broker tuning#

num.network.threads=8
num.io.threads=16                      # roughly 2x disk count
num.replica.fetchers=4                 # parallelism for replication
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
log.retention.check.interval.ms=300000
# Never elect an out-of-sync replica as leader: it truncates acknowledged data.
unclean.leader.election.enable=false
auto.create.topics.enable=false        # topics via CI, not by typo

auto.create.topics.enable=false prevents a whole class of confusion. With it on, a typo in a topic name silently creates a new topic with default settings — wrong partition count, wrong replication factor — and the producer succeeds while nobody ever consumes it.

Kafka relies heavily on the OS page cache, so leave most of the machine's memory to it rather than giving Kafka a large heap. 6GB of JVM heap with the rest as page cache is a typical shape.

Geo-replication#

clusters=primary,dr
primary.bootstrap.servers=kafka-eu:9092
dr.bootstrap.servers=kafka-us:9092
 
primary->dr.enabled=true
primary->dr.topics=orders|payments|inventory
# Replicates committed offsets so consumers can resume in the DR cluster
# roughly where they stopped, rather than from the beginning.
primary->dr.emit.checkpoints.enabled=true
primary->dr.sync.group.offsets.enabled=true
replication.factor=3

MirrorMaker 2 replicates topics, configuration and consumer offsets between clusters. By default it prefixes replicated topics with the source cluster name, which prevents loops in active-active setups but means consumers in the DR cluster subscribe to primary.orders rather than orders — worth knowing before the failover rather than during it.

Replication is asynchronous, so a failover loses whatever had not replicated. Measure the replication lag and treat it as your actual RPO.

What to take away#

Size partitions for parallelism and add headroom, because increasing them later breaks key ordering. Alert on under-replicated partitions and controller count immediately, and on consumer lag that is both large and growing. Disable auto topic creation and unclean leader election. And practise the offset reset before you need it in an incident.