01What is the crawl FOR — search index, LLM training data, archival? It sets size and freshness.
Fetch pages starting from seeds and store raw content for downstream use — the consumer defines what "done" means.
Free full guide
Web Crawler
Design a web-scale crawler that politely fetches and indexes billions of pages, deduplicates content, and keeps the corpus fresh.
00
Practice checkpoints
The interview rhythm stays compact, so the page can spend attention on the actual design decisions.
- 01Clarify scope
- 02Requirements + scale
- 03API + data model
- 04Draw architecture
- 05Deep dive
- 06Trade-off decision
01
Requirements that shape the design
Do not only state requirements. Ask for them. Each card pairs the design constraint with a clarification question you can say out loud before drawing the architecture.
Functional requirements
02How does the crawl grow — do extracted links feed back in?
Every fetched page is parsed for links, which are filtered and fed back into the frontier — the crawl sustains itself from the seeds.
03Politeness: requirement or nice-to-have?
A requirement: obey robots.txt (including crawl-delay), identify honestly via user-agent, and never overload any single host.
04Can operators inject seeds and priority URLs mid-crawl?
Yes. Operators can push URLs in at high priority, ahead of the links the crawl finds on its own. A crawl can be steered, not just left to run.
05What exactly do we store - raw bytes, parsed text, or both?
Both. We keep the raw content in blob storage, plus the extracted text and metadata. Reprocessing later must never mean fetching the page again.
06How fresh must robots.txt be?
Fresh within hours: a host must never be crawled on robots.txt rules more than a few hours old — acting on a stale allow is a compliance risk.
Out of scopeSearch ranking and indexing (the crawler produces the corpus, not the index) · JavaScript rendering of dynamic pages · Continuous re-crawling for freshness (single full crawl first)
Non-functional requirements
01How many pages, and how fast?
10B pages in under 5 days — that single line dictates fleet size and queue throughput.
02What protects the websites we crawl?
Honor each host's crawl-delay no matter the backlog: a million queued URLs for one site must drain slowly, never as a flood.
03The same page often lives at many URLs — mirrors, tracking parameters, www variants. Should the corpus keep each page only once?
Never fetch the same URL twice, and the same page reached through different URLs must land in the corpus only once.
04If a machine crashes partway through the crawl, can we lose the URLs it was working on, or must every URL eventually be accounted for?
Nothing is silently lost: every URL is eventually fetched or explicitly recorded as failed — a machine crash must never make work disappear.
05What about infinite URL spaces?
Depth caps, per-domain page budgets, and URL-pattern filters keep calendar pages and ad farms from consuming the crawl.
Keep asking — the interview is a conversation
Real interviews probe far more than a tidy list. These are the scope questions that separate candidates who interrogate the problem from those who recite it.
- How long must the fetched corpus be retained, and must we honor a site owner's takedown or deletion request?
- Is there a bandwidth or dollar budget for the crawl, or is the 5-day deadline the only constraint?
- Do we need an audit trail of what was fetched when — enough to prove we honored robots.txt if a site owner complains?
- Do we render JavaScript, or fetch raw HTML only?
- Is one full crawl enough, or does the corpus need continuous refresh?
02
Numbers that force architecture decisions
Treat every estimate as a pressure that justifies a component: cache, queue, partition, replica, worker pool, or fallback path.
Required fetch rate
10B pages ÷ 5 days10,000,000,000 ÷ 432,000 s ≈ 23K pages/s sustained
This is a fleet, not a server — and the frontier must hand out 23K polite URLs per second.
Fleet size
~2 s average fetch latency per page, so ~0.5 pages/s per connection · ~4K useful concurrent connections per box23K ÷ (4,000 × 0.5) ≈ 12 machines sustained — provision ~2× for retries and slow hosts
A couple dozen fetchers meet the deadline; the bottleneck is politeness, not compute.
URL-seen memory
10B URLs in a Bloom filter at 1% false positives (~10 bits each)10B × 10 b ≈ 12 GB — vs ~400 GB for exact strings
What matters is the Bloom answer "definitely new." A false positive just skips one URL. A false negative never happens, so nothing is ever crawled twice.
Storage for the corpus
10B pages × ~100 KB of stored HTML — the multi-MB quoted page weight is mostly images and scripts this crawler never fetches≈ 1 PB raw
Blob storage with compression; metadata (hashes, URLs) stays queryable in a separate store.
DNS pressure
23K fetches/s each needing resolutionwithout caching: 23K lookups/s · with per-fetcher DNS cache: ~1 per new host
An in-fleet DNS cache is mandatory — public resolvers would rate-limit the crawl to death.
Decision example
The numbers
Twenty-three thousand pages a second for five days straight. But the real constraint runs the other way: no single website may ever feel more than a trickle.
My choice
I would split the frontier into two stages. Front queues order URLs by priority; back queues give each host its own queue, with a token bucket per host that honors crawl-delay. Fetchers lease URLs instead of deleting them, ack on success, and let lease timeouts recover from crashes. A URL passes a Bloom-filter seen-check before enqueue, and after fetch a content hash catches the same page reached through different URLs. Robots rules cache per domain with a TTL, and DNS gets its own cache inside the fleet.
Avoid
What I would NOT do: use one global priority queue. The moment a big site dumps a million URLs, fetchers hammer that host and the crawl becomes a DDoS. I also would not track seen URLs in an exact database table. That means 400 GB of strings and a lookup per enqueue, when a 12 GB Bloom filter answers "definitely new" for free. Its rare false positive merely skips a URL, which is the safe direction to miss.
Change if
If the corpus needs continuous freshness instead of one crawl, the frontier gains a re-crawl scheduler: pages re-enter by change frequency and importance, and the seen-filter switches to a structure that supports aging out.
03
Architecture path
One complete picture first, then each path as its own diagram — the write path and the read path carry different traffic and justify different components.
Complete picture
Overview — every component
- Extracted links loop from the parser back into the frontier — the crawl feeds itself from its own discoveries.
- Inside the frontier, every host gets its own queue, and a URL is handed out only at that host's allowed pace.
- Politeness is built into the structure that hands out URLs — no single fetcher can flood a site, even by accident.
Path 1
Fetch path — polite by construction
A URL is only handed out when its host has a token; the lease returns to the queue on crash, retries back off, and repeated failures land in a dead-letter queue.
Path 2
Discovery path — the loop that feeds itself
Every page yields links; filters drop traps and junk, the Bloom filter drops everything already seen, and the survivors re-enter the frontier.
04
API and data model
Before optimizing, make the contract inspectable: endpoints, entities, ownership, retries, and state.
QUEUEfrontier.enqueue(url, depth)
resaccepted · dropped (seen / filtered / over budget)
Internal contract, not REST — a crawler has no public API. Enqueue passes the URL-seen filter and pattern filters first.
QUEUEfrontier.lease(fetcher_id) → CrawlTask
restask with lease TTL · ack(task) on success · nack → retry with backoff
Per-host back-queues enforce politeness: a fetcher only receives a URL when that host’s token bucket has a token.
GEThttps://{host}/robots.txt (external)
resrules cached in DomainState with TTL
The one external contract: obey Disallow and crawl-delay, and send an honest User-Agent.
Core entities
CrawlTaskurl · domain · depth · status: queued/leased/done/failed · retries
The unit of work; leased (not deleted) while a fetcher works on it, so crashes self-heal.
DomainStatedomain (PK) · robots_rules · crawl_delay · last_fetch_at · pages_crawled
Robots rules are cached per domain with a TTL; the token bucket lives here.
Pageurl · content_hash (SHA-256) · fetched_at · blob_ref
content_hash powers the second dedup layer — identical content under different URLs stores once.
05
Deep dive directions
Pick one lane for the final third of the interview. Each lane gives you the topic, the interviewer question it should answer, and the failure mode to avoid.
A million URLs, one small site
Ask
The frontier holds a million URLs for one modest website. What guarantees it never gets hammered?
Answer
All those URLs go into one queue for that host. A token bucket tied to the site's crawl-delay releases them one at a time, and fetchers can only take what the bucket grants. Add a thousand machines and that host still sees the same trickle — the queue enforces it, not goodwill.
Avoid
Rate-limiting inside fetchers — politeness must be structural, in the per-host queue, or a scaled-out fleet breaks it.
Have I seen this URL
Ask
Ten billion URLs — how do you answer "seen before?" per enqueue, in memory, and what does a Bloom false positive cost here?
Answer
Check a Bloom filter before every enqueue. Ten billion URLs at about 10 bits each fits in roughly 12 GB, versus 400 GB for exact strings. It never gives a false negative, so nothing is crawled twice. A rare false positive just skips one URL — the safe direction to miss.
Avoid
Treating the false positive as an error — skipping one URL is the designed cost; crawling twice is the failure.
Same page, different URL
Ask
Mirrors, tracking parameters, and www/non-www all serve identical content. Where does the second dedup layer sit, and on what key?
Answer
The second layer sits after fetch, keyed on a content hash. Hash the page body and look it up in a seen-hashes store. On a match, keep one canonical copy and record the other URL as an alias. Normalization handles tracking parameters and www variants; only the hash catches identical pages from different hosts.
Avoid
URL normalization alone — only a content hash catches true duplicates across hosts.
The infinite calendar
Ask
A site generates a valid "next month" link forever. What bounds the crawl, and how do you detect the trap?
Answer
Three bounds stack: a depth cap, a per-domain page budget, and URL-pattern filters that flag machine-made shapes like ever-incrementing dates. The budget does the real detecting. When a small domain burns through thousands of near-identical pages, it gets throttled or cut off — so no single trap can eat the crawl.
Avoid
Relying on depth alone — per-domain budgets and URL-pattern heuristics have to back it up.
A fetcher dies holding 4,000 URLs
Ask
One machine crashes mid-fetch. What happens to its in-flight work, and what does the recovery cost?
Answer
Nothing is lost, because each URL was leased, not deleted. When the dead machine stops acking, its 4,000 leases time out and the URLs go back to the queue for other fetchers. Recovery costs only the lease-timeout delay plus refetching what was in flight — bounded and cheap.
Avoid
Deleting a URL from the queue at hand-out time — lease with a timeout, ack on completion.
Ready to practice?
Talk through Web Crawler out loud and get AI scoring on the explanation.