💬
✉️
📸
▶️

Loading…

Design Google Search: crawling, indexing, and ranking

Google SearchGuided Mode

Web Crawl to Inverted Index Pipeline

Request Flow

1

Crawl Scheduler controls the fleet

The Crawl Scheduler maintains a priority queue of billions of URLs. Priority = f(PageRank, time-since-last-crawl, change frequency). High-authority news sites are crawled every few minutes; static pages monthly.

2

Crawler fetches and Content Processor extracts

The Web Crawler fetches HTML, respects robots.txt, and forwards raw content to the Content Processor. The processor parses the DOM, removes boilerplate, runs SimHash deduplication, and emits clean token documents to Kafka.

3

Inverted Index Builder hot-swaps segments

The Inverted Index builder builds new compressed index segments and hot-swaps them into the live Search Index Shard with zero downtime. The query path never waits for index builds.

Key Insight

The crawl-to-index pipeline is fully asynchronous. The query path reads from a static snapshot of the index; the crawl pipeline writes new snapshots in the background. Kafka decouples them completely.

Step 3 of 4