![Live streaming blocks
RTMP
TS + ?
(h264+AAC)
h264
AAC
Nxh264
NxAAC
TS
fMP4
Txt
XML
HLS
DASH
HTTP[s]
HLS
DASH
HTTP[s]](https://image.slidesharecdn.com/20190618-livestreamingatscale1-190629170011/85/mmsys2019-live-streaming-at-scale-7-320.jpg)
mmsys2019 live streaming at scale
- 1. LIVE STREAMING AT SCALE Jordi Cenzano | Director of engineering jcenzano@brightcove.com jordicenzano mmsys2019 2019/06/19
- 2. Agenda • Simple live streaming • Real (complex) live streaming blocks • Some challenges ◦ Ingest ◦ Personalized live streams (SSAI) ◦ Monitoring ◦ Latency ◦ Redundancy • Future • Conclusions
- 3. WHAT IS LIVE STREAMING?
- 5. Simple live streaming demo WebServer (nodeJS): https://gist.github.com/jordicenzano/a943bda656ad1e0a0568fbaad311af05 RTMP + transcode + HLS (ffmpeg): https://gist.github.com/jordicenzano/c5516b5854eb046bd1c27076fe4ba917 #!/usr/bin/env bash ffmpeg -hide_banner -listen 1 -i "rtmp://0.0.0.0:1935/live/stream" -c:v libx264 -b:v 900k -g 60 -profile:v baseline -preset veryfast -c:a aac -b:a 48k -hls_flags delete_segments -hls_time 6 -hls_segment_filename ~/480p_%05d.ts ~/480p.m3u8 #!/usr/bin/env bash http-server ~ -p 8080 --cors -c -1
- 6. WHAT IS REALLY LIVE STREAMING?
- 7. Live streaming blocks RTMP TS + ? (h264+AAC) h264 AAC Nxh264 NxAAC TS fMP4 Txt XML HLS DASH HTTP[s] HLS DASH HTTP[s]
- 8. LIVE INGEST
- 9. Live ingest From: https://en.wikipedia.org/wiki/Internet_service_provider • Data loss • Latency • Jitter • Reordering • Corruption • Duplication S L A
- 10. Live ingest options: RTMP • Pros ◦ Reliable (TCP based) ◦ Popular •Cons (for live streaming) ◦ Only 1 video + 1 audio + data ◦ Connection oriented ◦ Retries consume BW ◦ Slow packet loss recovery ◦ No real time guaranteed
- 11. Live ingest options •Pros ◦ Can transport everything ◦ Not connection oriented ◦ Designed for live ◦ Very common in the broadcast world •Cons ◦ Not reliable ◦ Complexity
- 12. Live ingest RTMP vs SRT Source code: https://github.com/jordicenzano/live-streaming-ingest-advance d-simulation • Comparison RTMP vs SRT ◦ 5 % packet loss ◦ 50ms delay ◦ 11 Mbps BW ◦ Test stream • 8Mbps VBR (9Mbps max) • h264 1 keyframe every 3s
- 13. PERSONALIZATION (SSAI: Server Side ad insertion)
- 15. • Considerations ◦ One manifest per viewer = 1:1 • Same for content • Different for adbreaks ◦ Track every viewer in a “DB” Live stream personalization 1M CCU HLS 6s TD 2.5 Tbps(@2.5Mbps) 333K rps (1 m3u8 1 redir ts/fmp4) 166K r-w/s ◦ Ultimate solution: Edge?
- 16. Live streaming SSAI SCTE 35 SCTE35 info Session info Job info Ad-123c.mp4 ad-123c-0001.ts
- 18. Live streaming SSAI impressions 1M CCU 6 ads / adbreak 20 imp / ad 120 M imp / adbreak
- 19. MONITORING
- 20. • Considerations ◦ Measure all you can • Impact of logging / metrics ◦ All real time • Metrics • Logs ◦ Aggregate and alarm wisely ◦ Automate Monitoring
- 23. • LHLS (periscope approach) ◦ Chunk transfer + advanced segments Low latency “simple” demo Source code: https://github.com/jordicenzano/go-ts-segmenter https://github.com/mjneil/go-chunked-streaming-server (From Matthew Neil)
- 24. REDUNDANCY
- 26. Redundancy blocks #EXTM3U #Example of HLS REDUNDANT STREAMS #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000, RESOLUTION=720x480 http://ALPHA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000, RESOLUTION=720x480 http://BETA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000, RESOLUTION=1920x1080 http://ALPHA.mycompany.com/md/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000, RESOLUTION=1920x1080 http://BETA.mycompany.com/md/prog_index.m3u8
- 27. Redundancy blocks: Full sync approach
- 28. Redundancy blocks: non-sync approach ... #EXTINF:4.000000, from-A-0000.ts #EXTINF:4.000000, from-A-0001.ts #EXT-X-DISCONTINUITY #EXTINF:4.000000, from-B-0001.ts .... A B
- 29. Future • Do more at the edge with multi CDN • Personalized streams ◦ Cohorts approach (hotstar 2019) • Low latency deployments ◦ ABR • Codecs ◦ More, faster, deeper pixels ◦ Context aware encoding
- 30. Conclusions: Experience based • Everything fails, and we are live ◦ Plan B, plan C, ,,,, plan Z ◦ Better hot - hot • Be sure the involved 3rd party are ready ◦ CDNs, AdServers • Load test (not easy) ◦ There are limits everywhere ◦ Individually ◦ End to end • Enjoy the challenge
- 31. Q & A