Message brokers: queues vs logs

Why it matters

"Use a queue" or "publish an event" is only half a decision; the broker you pick determines what is possible. Some brokers delete a message once it is consumed; others keep it for days and let any number of consumers read it from any point. Some give you per-message routing and acknowledgement; others give you ordered partitions and massive throughput. Choosing the wrong family means fighting the tool, and interviewers want to hear which one you mean and why.

The idea

A message broker receives messages from producers, stores them, and delivers them to consumers. The two dominant designs differ in what "stores" means.

Queue-based brokers

RabbitMQ, ActiveMQ, Amazon SQS, Google Cloud Tasks. A message is pushed to or pulled by a consumer, acknowledged, and deleted. The broker tracks per-message state, supports rich routing (exchanges, bindings, urgency levels, delays), and redelivers unacknowledged messages. Retention is "until consumed". They excel at task distribution and command-style messaging, where each message is a unit of work to be done once.

Log-based brokers

Apache Kafka, Apache Pulsar, Amazon Kinesis, Redpanda. A topic is split into partitions, each an append-only ordered log on disk. Messages are kept for a configured retention (hours to forever) regardless of consumption. Consumers do not acknowledge messages; each consumer group tracks an offset per partition, the position it has read up to. Many groups read the same log independently, and any group can rewind. Throughput is very high because the broker does little per message.

Partitions, keys and ordering

A partition is the unit of ordering and parallelism. Messages with the same key go to the same partition, so all events for one user or one order arrive in order. Within a consumer group, each partition is read by exactly one consumer instance, so the maximum parallelism of a group is the partition count. A hot key (one user generating most traffic) makes a hot partition, which is the classic scaling problem; the fix is a better key or more partitions, and repartitioning changes the key-to-partition mapping, so choose the count with headroom.

	Queue-based (RabbitMQ, SQS)	Log-based (Kafka, Pulsar, Kinesis)
Message lifetime	Until acknowledged	Retention period, independent of reads
Multiple consumers of one message	Fan-out via separate queues or exchanges	Native: every consumer group reads the whole log
Replay	Not possible once consumed	Rewind an offset, or start a new group from the beginning
Ordering	Per queue, loosely with many consumers	Strict per partition, by key
Per-message features	Urgency levels, delays, routing rules, per-message ack	Minimal; batching and offsets
Throughput	Tens of thousands per second per node	Millions per second per cluster
Operational weight	Lighter; SQS is zero-ops	Heavier; managed offerings help
Fits	Background jobs, task queues, request-reply, delayed work	Event streams, change data capture, analytics pipelines, event sourcing, multi-consumer integration

Choosing

Ask three questions. Will more than one consumer need each message, now or later? Log-based. Will anyone need to replay history (backfill a new service, recover from a consumer bug)? Log-based. Is this a pile of independent jobs where each should be done once, possibly with delays or urgency levels? Queue-based. Many systems run both: Kafka for domain events and streams, a queue for job distribution to workers, sometimes fed from a Kafka consumer.

Operating either

Queue-based: watch depth and age, set visibility timeouts and dead-letter queues, mind per-message overhead at high volume. Log-based: watch consumer lag (how far offsets trail the head) per group and partition, size partitions for peak parallelism, pick keys to avoid hot partitions, and set retention to cover the longest plausible replay or outage. Both: schema registry, correlation ids, idempotent consumers.

Trade-offs

Queue brokers are simpler, richer per message and forgetful; log brokers are durable, replayable, ordered and fast, and heavier to run and reason about. Retention and replay are what turn a message system into a source of truth other teams can build on, which is why log brokers have become the backbone of event-driven architectures and why they are overkill for a background job runner.

In an interview

Name the family and the reason: "Domain events go to Kafka, keyed by order id for per-order ordering, retained seven days so a new consumer can backfill and a buggy one can replay. Background jobs like email go to SQS with a dead-letter queue, because each is done once and SQS needs no operations." If asked about scale: "Partition count sets consumer parallelism; we watch consumer lag per group and alert when it grows."