πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Design YouTube: video upload, transcoding, and CDN

YoutubeGuided Mode

Queue: Decouple Upload from Processing

When a video lands in S3, an S3 event automatically enqueues a processing job. The uploader's request has already returned β€” transcoding happens asynchronously in the background via a Queue.

Request Flow

1

S3 event fires

When the uploader's PUT to S3 completes, S3 triggers an event (S3 ObjectCreated) that is forwarded to the Queue automatically.

2

Job enqueued

The queue holds the job until a processing worker picks it up. If all workers are busy, the queue buffers the job β€” no video is lost during spikes.

3

Upload returns immediately

The uploader sees 'βœ“ Upload complete' instantly. Transcoding begins in the background. The queue provides temporal decoupling β€” upload and processing are independent.

Why We Need This

Without a queue, the upload handler would need to wait for transcoding to finish before returning β€” potentially minutes. With a queue, the upload returns in milliseconds and transcoding proceeds at its own pace.

Key Insight

The queue is a buffer between upload rate and processing capacity. During a viral upload spike, the queue absorbs thousands of jobs while workers steadily drain it. Without the queue, your upload API would block waiting for overloaded transcoding workers.

Step 4 of 9