πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Design Instagram: media upload, CDN, and feed ranking

InstagramGuided Mode

Instagram's Media Pipeline: Resize, Compress, Transcode

Instagram generates 5 variants of every photo (thumbnail, low, standard, high-res, original). A Media Processing Pipeline triggered by S3 events handles this asynchronously via a Queue β€” the upload response returns instantly while workers process in parallel.

Request Flow

1

S3 fires upload event

When the client's PUT completes, S3 / Blob Storage triggers an event notification (S3 Event β†’ SNS β†’ SQS). The upload response to the client was already sent β€” this is fully async.

// S3 Event Notification
{
  event: 's3:ObjectCreated:Put',
  bucket: 'raw-uploads',
  key: 'user123/photo456.jpg'
}
2

Queue buffers the job

The Queue decouples S3 event bursts from pipeline capacity. If 10,000 photos upload simultaneously, the queue absorbs the spike while workers process at steady throughput.

3

Pipeline processes media

The Media Processing Pipeline dequeues a job and runs the processing chain: decode β†’ resize to multiple resolutions (thumbnail/medium/full) β†’ compress with libvips β†’ write outputs back to S3 / Blob Storage.

// Processing chain
original β†’ resize(1080Γ—1080) β†’ compress(85%)
         β†’ resize( 320Γ— 320) β†’ compress(80%)
         β†’ resize(  64Γ—  64) β†’ compress(75%)
4

Outputs written to S3

Multiple output files (thumbnail, medium, full) are stored with structured keys in S3 / Blob Storage. The Database record is updated to 'processed' status when all variants are ready.

Why We Need This

Processing media synchronously in the upload path would add 800ms–3s to every upload response. Async pipelines keep uploads fast and let processing scale independently.

Queue depth is a real-time signal of pipeline health. If depth grows faster than it drains, you autoscale workers. This elasticity is only possible with the queue as the buffer.

Key Insight

Never do heavy compute in the upload hot path. Accept the raw file instantly, acknowledge the client, then process asynchronously. The queue is your shock absorber between bursty uploads and steady-throughput workers.

Step 4 of 7