Design Dropbox: file chunking, sync, and storage
Dropbox
Guided ModeQueue: Decouple Upload from Sync Processing
At scale, the Metadata Service and Sync Service can't keep up with upload bursts synchronously. Adding a Message Queue between the API and processing pipeline buffers the load and decouples upload latency from sync propagation latency.
Request Flow
API enqueues sync event
After writing chunks to S3, the API publishes a lightweight event to the queue (SQS / Kafka). The upload response is returned to Device A immediately — sync happens asynchronously.
{
"type": "file_updated",
"user_id": "u123",
"file_id": "f456",
"s3_keys": ["chunk-a1b2", "chunk-c3d4"]
}Metadata Service consumes
The Metadata Service polls the queue, processes each event, and updates the Metadata DB. If it's overloaded, messages wait — safe, ordered, and durable.
Sync Service notifies
After the Metadata DB is updated, the Sync Service pushes the delta to Device B's long-poll connection.
Why We Need This
Without a queue, a sudden burst of 100,000 uploads would create 100,000 simultaneous writes to the Metadata DB — causing lock contention, timeouts, and cascading failures. The queue absorbs the burst; consumers process at their safe throughput rate.
Key Insight
The queue is the buffer between upload speed (bursty) and processing speed (steady). Adding queue depth monitoring tells you exactly when to scale metadata processing workers.
Step 6 of 8
