Design Instagram: media upload, CDN, and feed ranking
Instagram's Media Pipeline: Resize, Compress, Transcode
Instagram generates 5 variants of every photo (thumbnail, low, standard, high-res, original). A Media Processing Pipeline triggered by S3 events handles this asynchronously via a Queue β the upload response returns instantly while workers process in parallel.
Request Flow
S3 fires upload event
When the client's PUT completes, S3 / Blob Storage triggers an event notification (S3 Event β SNS β SQS). The upload response to the client was already sent β this is fully async.
// S3 Event Notification
{
event: 's3:ObjectCreated:Put',
bucket: 'raw-uploads',
key: 'user123/photo456.jpg'
}Queue buffers the job
The Queue decouples S3 event bursts from pipeline capacity. If 10,000 photos upload simultaneously, the queue absorbs the spike while workers process at steady throughput.
Pipeline processes media
The Media Processing Pipeline dequeues a job and runs the processing chain: decode β resize to multiple resolutions (thumbnail/medium/full) β compress with libvips β write outputs back to S3 / Blob Storage.
// Processing chain
original β resize(1080Γ1080) β compress(85%)
β resize( 320Γ 320) β compress(80%)
β resize( 64Γ 64) β compress(75%)Outputs written to S3
Multiple output files (thumbnail, medium, full) are stored with structured keys in S3 / Blob Storage. The Database record is updated to 'processed' status when all variants are ready.
Why We Need This
Processing media synchronously in the upload path would add 800msβ3s to every upload response. Async pipelines keep uploads fast and let processing scale independently.
Queue depth is a real-time signal of pipeline health. If depth grows faster than it drains, you autoscale workers. This elasticity is only possible with the queue as the buffer.
Key Insight
Never do heavy compute in the upload hot path. Accept the raw file instantly, acknowledge the client, then process asynchronously. The queue is your shock absorber between bursty uploads and steady-throughput workers.
Step 4 of 7
