💬
✉️
📸
▶️

Loading…

Design Dropbox: file chunking, sync, and storage

DropboxGuided Mode

Queue: Decouple Upload from Sync Processing

At scale, the Metadata Service and Sync Service can't keep up with upload bursts synchronously. Adding a Message Queue between the API and processing pipeline buffers the load and decouples upload latency from sync propagation latency.

Request Flow

1

API enqueues sync event

After writing chunks to S3, the API publishes a lightweight event to the queue (SQS / Kafka). The upload response is returned to Device A immediately — sync happens asynchronously.

{
  "type": "file_updated",
  "user_id": "u123",
  "file_id": "f456",
  "s3_keys": ["chunk-a1b2", "chunk-c3d4"]
}
2

Metadata Service consumes

The Metadata Service polls the queue, processes each event, and updates the Metadata DB. If it's overloaded, messages wait — safe, ordered, and durable.

3

Sync Service notifies

After the Metadata DB is updated, the Sync Service pushes the delta to Device B's long-poll connection.

Why We Need This

Without a queue, a sudden burst of 100,000 uploads would create 100,000 simultaneous writes to the Metadata DB — causing lock contention, timeouts, and cascading failures. The queue absorbs the burst; consumers process at their safe throughput rate.

Key Insight

The queue is the buffer between upload speed (bursty) and processing speed (steady). Adding queue depth monitoring tells you exactly when to scale metadata processing workers.

Step 6 of 8