Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
Download as PPTX, PDF3 likes1,572 views
The document details the architecture and performance characteristics of a 200 Gbit/s video platform, focusing on various components like balancers, edge servers, and video managers, along with their configurations and optimizations. It discusses the load profiles, bandwidth specifications, and challenges faced with p2p architecture and playback issues. Additionally, it highlights future directions for offloading traffic and improving client compatibility with emerging technologies.
Как построить видеоплатформу на 200 Гбитс / Ольховченков Вячеслав (Integros)
- 1. Building 200 Gbit/s video platform Vyacheslav Olhovchenkov
- 2. Load profile • Peak bandwidth 210Gbit/s • Peak bandwidth per edge 40Gbit/s • Daily traffic 1PTB • Views per day 6M • RPS to ruby app: 1000 • RPD to ruby app: 60M
- 3. Core components • Balancers • Edge servers • Video manager • Encoders • Storage servers • Stats server • Player w/ adv.support: IMA; VPAID, VAST 2,3 • API
- 7. Balancer (features) • Geo targeting • Stats • Edge selection • Serving player iframe • Secured manifest transfer • URL signing • Advertising module
- 8. Balancer (configuration) • CPU: E3-1270 V2 @ 3.50GHz • RAM: 8GB • SSD • Ubuntu 14.04 • RoR, unicorn • PostgreSQL, redis
- 9. Balancer • 1000 RPS at peak • 75% CPU utilization
- 10. Balancer (tips & tricks) • PgSQL => Redis for hot data • ORM optimization joins, indexes, pure sql • Linux/nginx optimization net.core.somaxconn=65535 net.ipv4.tcp_fin_timeout = 25… • RoR optimization worker_processes = 24 timeout = 60
- 11. Edge — history. V1.0. 5Gbit/s • Dual E5-2620@ 2.00GHz • RAM: 32GB • 1x10Gbit Intel NIC • ZMIRROR: – 12 SATA 3TB – L2ARC (6SSD 256GB) • ZFS default settings
- 12. Edge (tuning 1) Reason for multiple writes? zfs set atime=off tank 8Gbit/s
- 13. Edge (tuning 2) • HDS, HLS: segment size 300-500K • Too many files • Prefetch isn’t working Concatenate segments Second NIC 12 Gbit/s
- 14. HDD performance total = seek + rotate + transfer Seek: 9ms (datasheet) Rotate: 7200rpm, 4.17 ms half turn Transfer: BlockSize/160MB/s Total: 13ms + transfer time 128KB: 0.8 ms => 71.4 IOPS, 9.1MB/s 1MB: 6.4ms => 53.8 IOPS, 55MB/s
- 15. Edge (tuning 3) • zfs set recordsize=1M ZFS larger block support Matthew Ahrens • HEAD: r274673 • STABLE: r27608 • vfs.zfs.prefetch_disable=1 • Limited by CPU: 20Gbit/s
- 16. Edge — configuration (CPU) • Xenon E3 vs E5 E3: 32GB RAM Limit • 40Gbit => 50K clients • 1MB/client => 50GB buffers • UP E5 vs DP E5 High end => high price Less PCIe slots
- 17. Edge — configuration (disk) SATA average transfer: • 160MB/s linear: 40/(.160*8) = 31 SATA HDD • 50MB/s random: 40/(.050*8) = 100 SATA HDD SSD cost: • x8 expensive $/TB • x4 less size
- 18. Edge — configuration (final) • CPU: 2x E5-2640v3 8 core @ 2.60GHz) • RAM: 128GB • NIC: Chelsio 40Gbit • HBA: LSI2308 + onboard • HDD: SATA x18 (storage) • SSD: 477GB x 8 (cache)
- 19. Edge — performance • 40 Gbit • 40-50K clients • 5Mpps
- 20. Edge — tips & tricks • No HT • Dedicated package for apps • Dedicated package for NIC • No NUMA
- 21. Warm content
- 22. Warm content (2)
- 23. Warm content (3)
- 24. 200Gbit
- 25. P2P (WebRTC) • Streamroot (Fr) • Hot content: wins 67% 0. 2.25 4.5 6.75 9. 13:00 15:00 17:00 19:00 21:00 23:00 P2P CDN
- 26. P2P Architecture • Viewer authenticates and initializes the connection with Integros load balancer • At the same time, viewer connects to Streamroot tracker and requests peers • Tracker sends back intelligently-selected list of peers • A P2P webRTC data channel is created among viewers to exchange segments INTEGROS LOAD BALANCER STREAMROOT TRACKER PEER POOL % SERVER WebRTC WebRTC WebRTC
- 27. P2P Issues and Current Status • Issue Aug 2015: Significant traffic offload followed by partial playback issues • Reason: Flash-JavaScript interface: – Javascript based WebRTC module – Callback is not error resilient – Fun fact: not documented anywhere in Flash • Solution: fixed by tweaking Flash-JS interface • Next step: HTML5 first (hls.js / Dash.js) to avoid Flash associated issues
- 28. P2P Future • 80% Offload • More clients TODAY 2016 Chrome YES YES Firefox YES YES Opera YES YES Edge BETA YES Safari NO ? iOS SDK NO YES Android SDK NO YES STBs BETA YES
- 29. Links https://www.youtube.com/watch? v=tujArcDozq0 (railsclub Ruby Meetup №2) https://events.yandex.ru/lib/talks/ 2694/ (ruBSD2104) slawa@integros.com