💬
✉️
📸
▶️

Loading…

Design Dropbox: file chunking, sync, and storage

DropboxGuided Mode

Chunking Service: Split Files for Parallel Upload

Even with S3, uploading a 2 GB file as a single PUT is risky — one network blip restarts everything. The Chunking Service splits files into fixed-size parts (typically 4–64 MB) and uploads them in parallel. A failed chunk means only that chunk re-uploads.

Request Flow

1

File split into chunks

The Chunking Service divides the file into fixed parts. Each chunk gets a content hash (SHA-256). If Dropbox already has a chunk with that hash, it skips the upload — instant deduplication.

chunk[0] = file[0..4MB]  hash=a1b2...
chunk[1] = file[4..8MB]  hash=c3d4...
...
2

Chunks upload in parallel

Each chunk is PUT to S3 independently. The Chunking Service uploads 6 chunks simultaneously. A 100 MB file with 4 MB chunks = 25 parts, done in ~4 seconds vs ~40 seconds sequentially.

3

Chunk manifest saved

The Database stores the ordered list of chunk hashes for this file. To reconstruct: fetch chunks in order and concatenate.

INSERT INTO file_chunks (file_id, idx, hash, s3_key)
VALUES ...

Why We Need This

Chunking gives you three superpowers: parallelism (multiple chunks at once), resumability (only failed chunks retry), and deduplication (same chunk across files = one S3 object). Dropbox deduplicated 2 billion files using this technique.

Key Insight

Content-addressed storage: chunks are stored by hash, not by filename. Two users uploading the same ISO file share the underlying chunks — zero extra storage cost.

Step 3 of 8