![- Cloud video streaming [1-5]
- Mathematical modelling (Jackson queue etc.) use QoS constraint
enable more efficient scale-in-out etc)
- Not sufficient for real deployment (anomaly, unexpected behavior)
- Not sufficient for NFV -> heterogeneous underlying hardware (entropy)
- Not Sufficient for MPEG DASH were client behavior is not standardized,
hence modelling user load difficult
- Google golden signals, CPU thread state, Off CPU analysis -> no automated
step mapping to Specific workload (domain knowledge needed), automated
mapping useful for function abstraction NFV
Prior art
[1] Wu et al., "CloudMedia: When Cloud on Demand Meets Video on Demand," in IEEE ICDCS, Minneapolis, 2011, pp. 268-277.
[2] Nan et al., "Optimal allocation of virtual machines for cloud-based multimedia applications," in IEEE MMSP Workshop, 2012, pp. 175-180.
[3] D. Niu et al., "Quality-Assured Cloud Bandwidth Auto-Scaling for Video-on-Demand Applications," in IEEE Infocomm, 2012.
[4] Y. Jin, Y. Wen, C. Westphal , "Optimal Transcoding and Caching for Adaptive Streaming in Media Cloud: An Analytical Approach,“
IEEE TCSVT, vol. 25, no. 12, pp. 1914 - 1925, December 2015.
[5] J. He, Y. Wen, J. Huang, D. Wu , "On the Cost-QoE Trade-off for Cloud-based Video Streaming under Amazon EC2’s Pricing Models,“
IEEE TCSVT, vol. 24, no. 4, pp. 669 - 680, September 2013.](https://image.slidesharecdn.com/mmsys-slideshare-intel-nokia-180801074516/85/Mmsys-slideshare-intel-nokia-10-320.jpg)
![Virtualized Media Processing Function
- Media, audio/video is a large percentage of the online traffic expected to continue in 5G
- Compute capabilities in network edge can improve video distribution [6] and [7]
- We consider a streaming function that can stream content using adaptive streaming using
HLS/DASH + captions and encrypt the content, all from a single media source based on
Unified Origin. NOTE: this is an Apache plugin
- In this paper the media processing function is deployed in the central cloud,
were it serves as origin server.
[6] Rufael Mekuria, Jelte Fennema, and Dirk Griffioen. 2016. Multi-Protocol Video
Delivery with Late Trans-Muxing. In Proceedings of the 2016 ACM on Multimedia
Conference (MM '16). ACM, New York, NY, USA, 92-96. DOI:
https://doi.org/10.1145/2964284.2967189
[7] https://www.unified-streaming.com/news/
finnish-telecom-leader-elisa-teams-unified-streaming-late-transmuxing](https://image.slidesharecdn.com/mmsys-slideshare-intel-nokia-180801074516/85/Mmsys-slideshare-intel-nokia-13-320.jpg)
![Experimental Setup
[6] Rufael Mekuria, Jelte Fennema, and Dirk Griffioen. 2016. Multi-Protocol Video Delivery
with Late Trans-Muxing. In Proceedings of the 2016 ACM on Multimedia Conference (MM
'16). ACM, New York, NY, USA, 92-96. DOI: https://doi.org/10.1145/2964284.2967189
[7] https://www.unified-streaming.com/news/finnish-telecom-leader-elisa-teams-unified-streaming-late-transmuxing
Load generator by
Citrix
Simusers requesting
Different contents
2 vCPU 4GB Ram
Origin VM (apache)
OpenStack Liberty release (KVM based)
KPI Mapping
Framework
(TBD later)
Telemetry agent
Storage &
Visualisation](https://image.slidesharecdn.com/mmsys-slideshare-intel-nokia-180801074516/85/Mmsys-slideshare-intel-nokia-15-320.jpg)
Mmsys slideshare-intel-nokia
- 1. Automated Profiling of Virtualized Media Processing Functions Using Telemetry and Machine Learning Rufael Mekuria(Unified Streaming), Michael J. McGrath (Intel), Victor Bayon-Molino(Intel), Vincenzo Riccobene(Intel), Christos Tselios(Citrix), Artem Dobrodub(Nokia), John Thomson (Onapp) ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 2. - Context: Emerging 5G and 5G Multimedia technologies - Research challenge - Contribution - TALE based profiling approach - Virtualized media processing function - Profiling using TALE and telemetry - KPI Mapping using telemetry and machine learning - Conclusion/Future work Summary/Overview ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 3. Context (1): Emerging 5G Network Technologies 5G RAN Cloudification Distribution Virtualized Network Functions Network Function Virtualization Management & Orchestration Massive Mimo Network slicing Millimetre Wave technologies (26+ GhZ) Multi-Access Edge Computing Virtualisation technologies 5G Radio Access 5G Cloud/NFV virtualized core ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands Small Cells WLAN + LTE Cloud Operating System e.g. hypervisor
- 4. Context(2): Advanced Media Services Content encryption Water Marked streams Personalized advertisement 360 degree High Dynamic Range Personalized video streaming Content security Emerging Formats (smarter pixels) Many pixels Point Cloud Light Field Personalized captions (language) Ultra HD 4K, 8K High Frame Rate Increased Bandwidth of 5G radio link -> more data virtualized 5G cloud native core -> Smarter network Smarter streaming Computer Vision 5G Transcode /transmux AR/MR ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 5. Edge cloud Edge cloud Core CloudAccess Network User equipment User equipment Scalable data center cloud infrastructure Aggregation Network 4G LTE LTE Broadcast 5G Wireless Virtualized Network User equipment Cloud-RAN Radio network information orchestration Radio network information MEC Cloud Regional cloudvEPC vIMS Virtualized network infrastructure Superfluidity: a flexible functional architecture for 5G networks Giuseppe Bianchi et al. Volume27, Issue9 TETT Journal Special Issue: 5GPPP Feature Issue September 2016 Pages 1178-1186 5G Context(3): Example edge converged cloud native 5G architecture
- 6. - Hardware independence (X86, ARM, GPU, FPGA) - Time independence - Scale independence scale from 1 to many users (millions) - Location independence - Reduce costly overprovisioning Broad range of technologies needed (NFV,MEC,SDN etc…): A core of advances in cloud/NFV technologies (hence our work) Context (4): Design goals of Superfluid 5G network https://www.unified-streaming.com/blog/5g-superfluidity-and-future-streaming-video ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 7. + Rashid Mjumbi et al. Network Function Virtualization: State of the Art and Research Challenges IEEE Communications surveys & tutorials, VOL. 18. NO 1. First Quarter 2016 http://www.etsi.org/ technologies-clusters/technologies/nfv Context (5): Some NFV pointers Difference NFV vs Cloud: service/function abstraction: VNF, per function optimization possible ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 8. Context (6) Virtualisation Stack Hypervisor type 1 Hypervisor type 2 Operating system level * Mark Croes Performance analysis of virtualized video streaming service Bachelor Thesis, University of Amsterdam June 2017 ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia Our focus is type 1 virtualisation
- 9. R1: Performance modelling/work load characterization -> virtual box in the cloud vs. physical box, underlying hardware is heterogeneous, cloud operating system stack R2: Reduce overprovisioning of underlying physical/virtual infrastructure R3: KPI Mapping for more efficient telemetry/monitoring, identifying most Important metrics relating to service quality defined in SLA Goal: “efficient carrier grade cloud native processing functions” this work is a step in this direction Research Challenges ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 10. - Cloud video streaming [1-5] - Mathematical modelling (Jackson queue etc.) use QoS constraint enable more efficient scale-in-out etc) - Not sufficient for real deployment (anomaly, unexpected behavior) - Not sufficient for NFV -> heterogeneous underlying hardware (entropy) - Not Sufficient for MPEG DASH were client behavior is not standardized, hence modelling user load difficult - Google golden signals, CPU thread state, Off CPU analysis -> no automated step mapping to Specific workload (domain knowledge needed), automated mapping useful for function abstraction NFV Prior art [1] Wu et al., "CloudMedia: When Cloud on Demand Meets Video on Demand," in IEEE ICDCS, Minneapolis, 2011, pp. 268-277. [2] Nan et al., "Optimal allocation of virtual machines for cloud-based multimedia applications," in IEEE MMSP Workshop, 2012, pp. 175-180. [3] D. Niu et al., "Quality-Assured Cloud Bandwidth Auto-Scaling for Video-on-Demand Applications," in IEEE Infocomm, 2012. [4] Y. Jin, Y. Wen, C. Westphal , "Optimal Transcoding and Caching for Adaptive Streaming in Media Cloud: An Analytical Approach,“ IEEE TCSVT, vol. 25, no. 12, pp. 1914 - 1925, December 2015. [5] J. He, Y. Wen, J. Huang, D. Wu , "On the Cost-QoE Trade-off for Cloud-based Video Streaming under Amazon EC2’s Pricing Models,“ IEEE TCSVT, vol. 24, no. 4, pp. 669 - 680, September 2013.
- 11. - Automated profiling of virtualized media processing functions to reduce overprovisioning in an NFV/Cloud deployment - Automated KPI mapping of virtualized media processing functions, enabling more efficient telemetry/targeted telemetry based monitoring (this can enable more efficient scale-in-out etc) - Emperical approach for cloud video streaming (we only focus on profiling and KPI mapping) Contribution ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 12. TALE metric collection summary Metrics are obtained from the cloud hardware: e.g. Intel SNAP, OpenStack Ceilometer, Amazon Cloudwatch etc…. Metrics obtained from OS/hypervisors, compute, storage, networks etc. Throughput Anomaly Latency Entropy Full Stack Monitoring: collect as many metrics from each of the layers, later then statistics/machine learning will be used to identify key metrics -> we discuss the analytics pipeline later ACM Multimedia Systems Conference 2018, June 12-15 Amsterdam the Netherlands: 5G Multimedia
- 13. Virtualized Media Processing Function - Media, audio/video is a large percentage of the online traffic expected to continue in 5G - Compute capabilities in network edge can improve video distribution [6] and [7] - We consider a streaming function that can stream content using adaptive streaming using HLS/DASH + captions and encrypt the content, all from a single media source based on Unified Origin. NOTE: this is an Apache plugin - In this paper the media processing function is deployed in the central cloud, were it serves as origin server. [6] Rufael Mekuria, Jelte Fennema, and Dirk Griffioen. 2016. Multi-Protocol Video Delivery with Late Trans-Muxing. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 92-96. DOI: https://doi.org/10.1145/2964284.2967189 [7] https://www.unified-streaming.com/news/ finnish-telecom-leader-elisa-teams-unified-streaming-late-transmuxing
- 14. Performance of Live vs. VoD - Performance analysis and planning model for live and VoD - Basic setup source -> origin -> CDN -> Client source origin CDN client load on origin for increasing number of users https://www.unified-streaming.com/blog/scaling-video-streaming-live-versus-vod Does not converge for VoD!
- 15. Experimental Setup [6] Rufael Mekuria, Jelte Fennema, and Dirk Griffioen. 2016. Multi-Protocol Video Delivery with Late Trans-Muxing. In Proceedings of the 2016 ACM on Multimedia Conference (MM '16). ACM, New York, NY, USA, 92-96. DOI: https://doi.org/10.1145/2964284.2967189 [7] https://www.unified-streaming.com/news/finnish-telecom-leader-elisa-teams-unified-streaming-late-transmuxing Load generator by Citrix Simusers requesting Different contents 2 vCPU 4GB Ram Origin VM (apache) OpenStack Liberty release (KVM based) KPI Mapping Framework (TBD later) Telemetry agent Storage & Visualisation
- 16. Profiling(1): Throughput tests (Tale) - User Arrival rate matters for stability! - Bitrate capped at 3.6 GBit
- 17. Profiling(2): Behavior tAle - Fix/configure apache configuration of mpm (multi processing threading module) - Result -> larger number connections and higher throughput 7Gbit (from 3.6 Gbps)
- 18. - Operational range leading to a breaking point - Previous results only showed saturation - How to identify/predict the breaking point ? Profiling(3): Throughput tests (Tale)
- 19. Profiling(4): Anomalous Behavior tAle - Interrupt storm dealing with interupts from I/O: NIC/storage - VM exits (inter process communication hypervisor and VM) (at least one exit and entry per interrupt) - Overwhelms the CPU - This is an example of virtualization overhead in hypervisor based virtualization, should be accounted for in practical systems
- 20. Profiling(5): VM Exits caused by interrupts Guest VM Hypervisor Interuptable ? No RFLAGS.IF=0 Set Interupt window exit Wants to send interupt Exit: reason interrupt window - Record cause of an exit in VM exit information - Save processor state: control reg. Debug reg., pending exception in guest state area - Save msr (machine specific registers) - Load processor based on host state - After control is completed a similar VM entry operation will happen Steps in dealing with interupts when guest VM is not interuptable
- 21. Profiling (7) Latency and Entropy summary taLE Single hardware/OS configuration -> hardware entropy not studied Other entropic behavior not observed in test setup -> e.g. time of day, temperature Analysis of subsystem metrics e.g. CPU utilization, memory etc. under normal operational conditions did not reveal any unexpected behaviours Undefined behavior after breaking point
- 22. KPI(1): Automated KPI Mapping KPI Mapping framework Analytics Pipeline Ensemble approach is taken combining the different algorithms, metrics with > 75% Machine learning approach with ensemble approach
- 23. KPI(2): KPI Mapping bootstrap Throughput per client Received bytes per client (MB/s) Client Latency per user request Delay to first byte received Client Transaction Failure rate Number of requests resulting in an error Client Requests / second requests handled per second per user Client Idea is to find the cloud metrics that can be measured by telemetry that correlate most to these client side KPI’s Client side metrics (measured by load generator)
- 24. KPI(3): KPI Mapping results non-optimized network media processing function: throughput and latency correlate to memory and network metrics throughput:network 98 % latency:memory 80% Optimized processing function: throughput and latency correlate to scheduler metrics proc_schedstat_wake_up_local:throughput local wakeup calls proc_schedstat_running:latency time running processes
- 25. Discussion highlights KPI/profiling p1. Massive telemetry combined with Machine Learning is useful for understanding virtaulized media processing function behavior p2. Operational range identified, behavior predictable, but collapses beyond Breaking point (not saturation) it will be important to detect catastrofic failure before it happens K1. non optimized KPI maps to subsystem metrics memory, network to Throughput/latency K2. scheduler metrics map for optimized function to throughput and latency K3. steady state and non steady state should be distinguished and detected K4. then in both states the respective KPI mapping can be used to avoid catastrofic failure K5. 85% reduction of metrics to be collected was achieved, by not collecting non relevant metrics K6. 80% to 95 accuracy for throughput
- 26. Conclusion / Future Work 1. Systematic automated approach for KPI mapping and profiling 2. Not based on pre-defined analytical models 3. Based on real production technologies (OpenStack, Unified Origin), results can be useful to enable KPI driven scaling, which can reduce overprovisioning 4. A small step in the goal “efficient carrier grade cloud native processing functions”
- 27. Thank You! Contact me: Rufael Mekuria rufael@unified-streaming.com Contact Vincenzo: vincenzo.m.riccobene@intel.com This work is supported by the European Union (H2020 RIA, GA No. 671566) Superfluidity.