"Kafka: queues and streams"

Kafka is the tool people name when a design needs a durable pipe between parts. It is used as a queue, as an event log, and as the backbone of stream processing. The model is simple, and the details are where answers get weak.

Crux: Kafka is a replicated, append-only log split into partitions. Order is guaranteed only within a partition, and a consumer group splits partitions among its members.

What it is and why it exists

A producer appends messages to a topic. Consumers read them. Unlike a classic queue, Kafka does not delete a message when it is read. The log is kept for a set retention time (7 days by default), and each consumer tracks its own position, called an offset. That gives you:

• Decoupling. Producers and consumers scale and fail on their own.
• Replay. A new consumer, or a fixed bug, can re-read old events.
• Many readers. Several independent consumer groups read the same events.
• Buffering. A burst is absorbed by the log while consumers catch up.

How it works, at the level you must explain

Topics, partitions and keys

A topic is split into partitions. Each partition is an ordered log on disk, stored on a broker, and replicated to other brokers.

• The producer picks the partition from the message key (a hash). All messages with one key go to the same partition, so they stay in order. With no key they spread out.
• Ordering is only promised inside one partition. If you need all events for one order to be processed in order, use the order id as the key.
• Partitions are also the unit of parallelism. A topic with 12 partitions can have at most 12 consumers in a group working at once.

Consumer groups and offsets

• A consumer group is a set of consumers sharing a topic. Each partition goes to one member of the group. Add consumers (up to the partition count) to scale. If a member dies, its partitions are handed to the others (a rebalance).
• The consumer commits offsets: "I have processed up to here." After a crash, it resumes from the last commit.
• A slow consumer builds up lag, the gap between the newest message and its offset. Lag is the metric to watch.

Replication and durability

Each partition has a leader and follower replicas, commonly 3 copies in total. The leader takes reads and writes, and followers copy it. The set of followers that are caught up is the in-sync replica set (ISR).

• Producer setting acks=all waits until the in-sync replicas have the message. Safer, slower.
• acks=1 waits for the leader only. A leader crash can lose the message.
• A common safe setup is 3 replicas, acks=all, and a minimum in-sync count of 2.

Kafka keeps cluster metadata with its own Raft-based controller in recent versions. Older versions used ZooKeeper for it.

Delivery guarantees

• At most once: commit the offset before processing. A crash loses the message.
• At least once: process, then commit. A crash means reprocessing, so duplicates are possible. This is the usual choice.
• Exactly once: Kafka offers idempotent producers and transactions, which give this inside Kafka's own read-process-write loop. For effects outside Kafka (an email, a payment) you still need idempotent consumers, for example by checking a unique event id.

Handling failure

• Retries with backoff for temporary errors.
• A dead-letter topic for messages that keep failing, so one bad message does not block a partition.
• Backpressure: consumers pull at their own pace. Watch lag and add consumers, or partitions, when it grows.

Sizing before scaling

Do the arithmetic first. Say a service emits 40,000 events per second, each about 1 KB. That is about 40 MB per second, which a small cluster handles with room. Kafka may be overkill for a tenth of that. If the figure is huge, divide by what one partition handles to get a minimum partition count, then add headroom. It is much easier to start with enough partitions than to add them later, because adding partitions changes which partition a key maps to, which breaks per-key ordering for existing keys. Too few partitions also cap how many consumers can work in parallel, so extra brokers or consumers sit unused.

Hot partitions

A key that is much busier than the rest overloads one partition. Example: an ad click stream keyed by ad_id, where one ad is shown during a final match and gets a thousand times the traffic.

• Do you need ordering for that key? If not, drop the key or use a random one.
• Salt the key: append a small random suffix (ad_9:0 to ad_9:7). Load spreads over eight partitions. The cost is that totals for the ad are now split, so a downstream job must combine the eight pieces.
• Compound key: use ad_id plus a second field that is naturally spread out (such as user bucket).
• Dedicated topic: move the few known hot keys to their own topic with more partitions.

Queue style or stream style

The same topic can be read in two ways. As a queue, one consumer group shares the work, each message handled once by the group. As a stream, several groups each read all messages for different purposes (billing, analytics, search). Say which one you mean.

Retries without a built-in retry

Kafka has no per-message retry or delay. A consumer that fails on message 41 must either retry in place (which blocks the partition) or push the message to a retry topic with a delay, then on to a dead-letter topic after a limit. If you need per-message delay, acknowledgement and retry as the main feature, a queue service is simpler.

Throughput levers

• Batching: the producer waits a few milliseconds (linger) to fill a batch. Bigger batches mean higher throughput and slightly higher latency.
• Compression of batches saves network and disk, for a little CPU.
• Writes are sequential appends and reads often come from the operating system cache, which is why Kafka is fast on ordinary disks.
• Retention is by time or size. Log compaction keeps only the latest message per key, which suits "current state" topics.

When to use it, trade-offs, numbers

• What: a durable, partitioned, replicated log.
• Use when: high-volume events, several independent consumers, replay or event sourcing, buffering bursts between services, and feeding stream processing and search indexes.
• Know: partitions, keys, ordering scope, consumer groups, offsets, replication and acks, at-least-once plus idempotent consumers.
• Careful: it adds latency compared with a direct call, it is work to run, and it is wrong in a path where a user waits for an answer.

Kafka or something else?

• SQS or RabbitMQ: a queue where each message is processed and removed, with per-message acknowledgement, delay and easy retry. Pick it for simple task queues with no need for replay. SQS is managed and simple.
• Redis streams or lists: light and fast, but memory-bound and with fewer durability guarantees.
• Pub/sub services: simple fan-out, often without replay.
• Writing to the database and polling: fine for low volume, and simplest.

Numbers worth quoting loosely: retention of days is the default, a single partition handles on the order of tens of megabytes per second, and a message is capped at about 1 MB by default. I am treating throughput figures as order of magnitude only, because they depend on hardware and settings. For large payloads, put the file in blob storage and send a reference.

Common mistakes

• Expecting global ordering across partitions.
• A low-cardinality or skewed key, which overloads one partition.
• Assuming exactly-once, then charging a card twice on a retry.
• More consumers than partitions, so some sit idle.
• Using it for a request that a user is waiting on.
• Putting big payloads in messages.

• Adding partitions late and breaking the key-to-partition mapping.
• Putting a hot key on a single partition and calling it scaled.
• Using it as a database or for large files. Store a reference to the file.
• Believing "Kafka never goes down". Brokers fail, leaders move, and a producer must handle errors.

Check yourself

• How is Kafka different from a queue that deletes messages on read?
• What does the message key control, and why does it matter for ordering?
• What is a consumer group, and what limits its parallelism?
• What happens when a consumer crashes after processing but before committing?
• What does acks=all buy you, and what does it cost?
• When would you choose SQS over Kafka?
• How would you spread the load of one very busy key, and what does it cost you?
• Why is it risky to add partitions to a live topic?
• How do you retry a failed message when Kafka has no retry feature?