Design Google Search: crawling, indexing, and ranking
Google Search
Guided ModeWeb Crawl to Inverted Index Pipeline
Request Flow
Crawl Scheduler controls the fleet
The Crawl Scheduler maintains a priority queue of billions of URLs. Priority = f(PageRank, time-since-last-crawl, change frequency). High-authority news sites are crawled every few minutes; static pages monthly.
Crawler fetches and Content Processor extracts
The Web Crawler fetches HTML, respects robots.txt, and forwards raw content to the Content Processor. The processor parses the DOM, removes boilerplate, runs SimHash deduplication, and emits clean token documents to Kafka.
Inverted Index Builder hot-swaps segments
The Inverted Index builder builds new compressed index segments and hot-swaps them into the live Search Index Shard with zero downtime. The query path never waits for index builds.
Key Insight
The crawl-to-index pipeline is fully asynchronous. The query path reads from a static snapshot of the index; the crawl pipeline writes new snapshots in the background. Kafka decouples them completely.
Step 3 of 4
