Design YouTube: video upload, transcoding, and CDN
Youtube
Guided ModeVideo Processing Pipeline: Transcode & Segment
The Video Processing Pipeline picks up jobs from the queue and runs each video through five stages: Ingest β Transcode β Segment β Thumbnail β Package. This work is asynchronous background processing, so the uploader's request path ends once the job is durably queued.
Request Flow
Transcode into resolutions
The Video Processing Pipeline re-encodes the raw video into 144p, 480p, 720p, 1080p, and 4K using codecs like H.264, VP9, and AV1. Each resolution is a separate output file.
// 5 resolutions Γ 3 codecs = 15+ output variants
transcode('raw.mp4', { codec: 'h264', height: 1080 })
transcode('raw.mp4', { codec: 'vp9', height: 720 })Segment into HLS chunks
Each resolution is split into 6-second .ts files (HLS) or 4-second .mp4 fragments (DASH). A 10-minute video becomes ~100 segment files per quality level β enabling adaptive streaming.
// HLS segmentation output master.m3u8 1080p/segment_000.ts (6s) 1080p/segment_001.ts (6s) 720p/segment_000.ts (6s)
Package manifest file
The final output is a manifest file (master.m3u8) that points to all quality variants and their segment URLs. The streaming client downloads this first and uses it to discover all options.
Why We Need This
Transcoding is hours of CPU work. Without async processing, the upload API would block for 30 minutes per video. With the queue-driven pipeline, uploads complete in seconds, the durable queue absorbs spikes, and a horizontally scaled processing fleet drains jobs in the background without extending request latency.
Key Insight
The processed segments are written back to a separate S3 prefix (processed/). Raw uploads are cheap, temporary storage. Processed segments are permanent, CDN-ready, and content-addressable by hash. In this guide the pipeline is modeled as asynchronous work, not part of the uploader's synchronous latency path.
Step 5 of 9
