Design YouTube: video upload, transcoding, and CDN
Youtube
Guided ModeQueue: Decouple Upload from Processing
When a video lands in S3, an S3 event automatically enqueues a processing job. The uploader's request has already returned β transcoding happens asynchronously in the background via a Queue.
Request Flow
S3 event fires
When the uploader's PUT to S3 completes, S3 triggers an event (S3 ObjectCreated) that is forwarded to the Queue automatically.
Job enqueued
The queue holds the job until a processing worker picks it up. If all workers are busy, the queue buffers the job β no video is lost during spikes.
Upload returns immediately
The uploader sees 'β Upload complete' instantly. Transcoding begins in the background. The queue provides temporal decoupling β upload and processing are independent.
Why We Need This
Without a queue, the upload handler would need to wait for transcoding to finish before returning β potentially minutes. With a queue, the upload returns in milliseconds and transcoding proceeds at its own pace.
Key Insight
The queue is a buffer between upload rate and processing capacity. During a viral upload spike, the queue absorbs thousands of jobs while workers steadily drain it. Without the queue, your upload API would block waiting for overloaded transcoding workers.
Step 4 of 9
