Design Dropbox: file chunking, sync, and storage
Dropbox
Guided ModeChunking Service: Split Files for Parallel Upload
Even with S3, uploading a 2 GB file as a single PUT is risky — one network blip restarts everything. The Chunking Service splits files into fixed-size parts (typically 4–64 MB) and uploads them in parallel. A failed chunk means only that chunk re-uploads.
Request Flow
File split into chunks
The Chunking Service divides the file into fixed parts. Each chunk gets a content hash (SHA-256). If Dropbox already has a chunk with that hash, it skips the upload — instant deduplication.
chunk[0] = file[0..4MB] hash=a1b2... chunk[1] = file[4..8MB] hash=c3d4... ...
Chunks upload in parallel
Each chunk is PUT to S3 independently. The Chunking Service uploads 6 chunks simultaneously. A 100 MB file with 4 MB chunks = 25 parts, done in ~4 seconds vs ~40 seconds sequentially.
Chunk manifest saved
The Database stores the ordered list of chunk hashes for this file. To reconstruct: fetch chunks in order and concatenate.
INSERT INTO file_chunks (file_id, idx, hash, s3_key) VALUES ...
Why We Need This
Chunking gives you three superpowers: parallelism (multiple chunks at once), resumability (only failed chunks retry), and deduplication (same chunk across files = one S3 object). Dropbox deduplicated 2 billion files using this technique.
Key Insight
Content-addressed storage: chunks are stored by hash, not by filename. Two users uploading the same ISO file share the underlying chunks — zero extra storage cost.
Step 3 of 8
