01Is this a task queue that deletes on delivery, or a log consumers replay?
A durable log: records are appended, kept by retention policy, and readable again from any offset — consuming never deletes anything.
Premium preview
Distributed Message Queue
Design a Kafka-style durable, partitioned log broker that ingests high-throughput event streams.
00
Practice checkpoints
The requirements are open as a taste. From the numbers onward, the full guide opens in the app.
- 01Clarify scope
- 02Requirements + scale
- 03API + data modelUnlocks in the app
- 04Draw architectureUnlocks in the app
- 05Deep diveUnlocks in the app
- 06Trade-off decisionUnlocks in the app
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02Does every consumer need messages in one global order?
No — order is promised per partition only: records with the same key land in the same partition in append order; global total order is explicitly not offered.
03Can two teams read the same stream without stealing each other’s messages?
Yes — every consumer group gets the full stream with its own offsets; inside one group each partition is owned by exactly one member, which is where parallelism comes from.
04A consumer crashes mid-batch — where does its replacement start?
At the group’s last committed offset: progress is committed explicitly by the consumer, so where you commit relative to processing IS the delivery guarantee.
05Do a payments topic and a metrics topic need the same durability?
No — replication factor and producer acks are set per topic: payments runs 3 replicas with acks=all; metrics can run acks=1 and accept rare loss for speed.
06How long do records live on the broker?
By policy per topic: delete past a time or size limit (say 7 days) for event streams, or compact — keep only the newest record per key — for changelog topics.
Out of scopeDelivery products built on top — webhooks, push notifications, email fan-out are clients of this broker, not the broker · Stream processing (windows, joins, aggregations) — that lives in consumer frameworks reading the log · Per-message routing rules and subscriber filters — consumers pick topics and partitions; the broker never inspects payloads
Non-functional requirements
01What ingest rate are we building for?
Hundreds of MB/s per topic sustained: sequential appends plus producer batching keep the disk writing at near-hardware speed, and throughput scales by adding partitions.
02The producer got its ack and the broker dies — is the record safe?
With acks=all and a minimum of 2 in-sync replicas, an acknowledged write survives leader death: the ack only fires after a follower also has the record.
03How long is a partition dark when its broker dies?
Broker failure pauses its partitions for seconds, not minutes: an in-sync follower is promoted and clients retry against the new leader with no acked data lost.
04What end-to-end latency is acceptable?
Tens of milliseconds p99 produce-to-consume under normal load — batching adds a few ms deliberately because it buys an order of magnitude in throughput.
05A consumer falls an hour behind — what happens to the broker?
A lagging consumer never destabilizes the broker: consumers pull at their own pace and the retained log is the buffer, so lag is an alerting metric, not broker pressure.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- Can consumers absorb duplicates with an idempotency key, or does the business truly need exactly-once and its throughput cost?
- Is ordering needed globally, per key, or not at all? That one answer decides the partitioning design.
- What are the ingest rate and the retention window — are we sizing this cluster for throughput or for storage?
- When a broker dies, do we pause its partitions for a few seconds, or stay up at the cost of acknowledged data (unclean election)?
- How far behind may consumers fall, and what should happen when lag approaches the retention window and unread data is about to expire?
01
Unlock the full playbook for Distributed Message Queue
Numbers, architecture diagram, API and data model, deep dives, expected topics, self-check, whiteboard starter, and common mistakes unlock inside the app.
Numbers that force architecture decisions
Locked in the app
Architecture path
Locked in the app
API and data model
Locked in the app
Deep dive directions
Locked in the app
+Series