Messaging Fundamentals & Patterns

Every broker — RabbitMQ, Kafka, SQS, Pub/Sub — implements the same handful of ideas with different words. Learning the ideas once means the next broker is a configuration exercise rather than a new subject.

The two delivery models#

The distinction maps onto a semantic one that is worth being explicit about.

A command says "do this" and has exactly one correct handler. Charging a payment twice because two consumers both picked it up is a bug. Commands belong on queues.

An event says "this happened" and any number of parties may care. When an order is placed, inventory wants to reserve stock, analytics wants to count it, and the notification service wants to email the customer. None of them should know about the others. Events belong on topics.

Getting this backwards produces recognisable pain. Commands on a topic cause duplicate side effects. Events on a queue mean adding a new consumer requires changing the producer, which defeats the entire point of using a broker.

Competing consumers#

Scaling a queue means adding consumers, and the broker round-robins messages between them. This is the standard way to increase throughput, and it costs you ordering: three consumers processing messages 1, 2 and 3 concurrently will finish in an unpredictable order.

When order matters — and it usually matters per entity, not globally — the answer is partitioning by key. All messages for order 4711 go to the same partition or the same consumer, so they are processed in sequence, while messages for different orders proceed in parallel. Kafka does this natively with the message key; RabbitMQ needs a consistent-hash exchange or single-active-consumer queues.

Global ordering across all messages requires a single consumer and gives up parallelism entirely. Before accepting that trade, check whether you actually need it. Most systems need per-customer or per-order ordering, which partitioning gives you at full throughput.

Acknowledgement#

The three modes and what each one costs you:

Distinguishing a transient from a permanent failure is the part that needs judgement. A database timeout is transient — requeue it. A payload that fails schema validation is permanent — no number of retries will make it parse, so send it straight to the dead-letter queue. Requeuing a permanent failure creates a poison message that loops forever, consuming capacity and blocking the messages behind it.

Dead letters#

A dead-letter queue is where messages go when they cannot be processed. It is not optional infrastructure, and it needs three things attached to it.

An alert on depth greater than zero. A DLQ nobody watches is a silent data-loss mechanism. Messages arrive there precisely because something needs human attention.

Enough context to diagnose. The message alone rarely explains the failure. Record the original destination, the failure reason, the exception, the attempt count and the timestamp — RabbitMQ adds x-death headers automatically; with Kafka you add them yourself when publishing to the DLT.

A replay path. Once the bug is fixed, you need to move messages back. Design this before you need it, because writing a replay tool during an incident is not when you want to be thinking about ordering and idempotency.

Delivery guarantees#

At-most-once means fire and forget. Fast, lossy, appropriate for data where the next sample supersedes the last.

At-least-once means the broker redelivers until acknowledged. Duplicates are guaranteed to happen eventually — during a rebalance, a network partition, or a consumer crash after processing but before acking. This is what nearly every system uses.

Exactly-once is not achievable as a delivery guarantee across an unreliable network; it is the two generals problem. What is achievable is exactly-once processing: at-least-once delivery plus a consumer that produces the same result whether it sees a message once or five times.

Idempotency has two implementations. Where the operation is naturally idempotent — setting a status to a fixed value, an upsert keyed by a business identifier — you need nothing extra, and designing for this is the cheapest option available. Where it is not, deduplicate on a message id with a unique constraint doing the atomicity, not a check-then-insert.

Patterns worth naming#

Request–reply turns messaging into RPC: the producer sends a correlation id and a reply-to address, the consumer answers on that address. Useful, but ask first whether you actually want synchronous behaviour — if you do, HTTP is simpler and easier to debug.

Content-based routing sends a message to different destinations based on its content. RabbitMQ does this natively with exchanges and routing keys; with Kafka you either publish to different topics or filter in the consumer.

Message TTL expires messages that are no longer useful. A price update from an hour ago is worse than no update, and expiring it — usually into a DLQ — is better than processing stale data.

Delayed delivery schedules a message for the future. Retry with backoff, "cancel the order if unpaid in 15 minutes", reminder emails. RabbitMQ has a delayed-message plugin; Kafka needs a separate scheduler or per-delay topics.

Claim check stores a large payload externally and passes a reference. Brokers are optimised for small messages, and a 50MB payload on a queue degrades everything sharing it.

Designing the message itself#

The message is a contract, and it outlives every deploy. Four properties make it survivable.

Give it a unique id so consumers can deduplicate, and a type so they can dispatch without guessing. Include a timestamp so late arrivals are detectable. And carry the correlation id from the originating request, so a trace does not stop at the broker.

Keep events self-contained enough to be useful without a callback, but not so fat that they duplicate another service's model. A middle ground that works well: include the identifier plus the few fields consumers demonstrably need, and let them fetch the rest.

Version from the start. Adding an optional field is safe; removing or renaming one is not. When a breaking change is unavoidable, publish a new event type alongside the old one and retire the old one on a schedule, exactly as you would with an HTTP API.

What to take away#

Commands go on queues, events go on topics. Acknowledge after processing, distinguish transient from permanent failures, and give every queue a dead-letter destination with an alert on it. Assume duplicates and make consumers idempotent — that, not a broker feature, is what gives you exactly-once processing.