IFIP Networking 2015
Download as PPT, PDF0 likes873 views
This document presents research on calibrating the DQX model for quality of experience (QoE) prediction in voice over IP (VoIP) calls. The researchers conducted experiments varying latency, packet loss, jitter, and bandwidth to collect user ratings. They calibrated the DQX model parameters to the data and compared it to other QoE models. The results showed that DQX is flexible but the influence factors require further tuning. Overall, DQX predicted mixed variable scenarios reasonably well compared to collected user ratings. Future work will explore additional variables, more test conditions and calibrating DQX for other services.
![© 2015 UZH,
VoIP-based Calibration of the DQX Model
Christos Tsiaras, Manuel Rösch, Burkhard Stiller
Department of Informatics IFI, Communication Systems Group CSG,
University of Zürich UZH
[tsiaras,stiller]@ifi.uzh.ch manuel.roesch@uzh.ch
IFIP Networking 2015, Toulouse, France, May 20, 2015
QoE Models for VoIP
DQX and Goals
Experiments and Results
Conclusion](https://image.slidesharecdn.com/ifinetworking2015-150519094741-lva1-app6891/85/IFIP-Networking-2015-1-320.jpg)
IFIP Networking 2015
- 1. © 2015 UZH, VoIP-based Calibration of the DQX Model Christos Tsiaras, Manuel Rösch, Burkhard Stiller Department of Informatics IFI, Communication Systems Group CSG, University of Zürich UZH [tsiaras,stiller]@ifi.uzh.ch manuel.roesch@uzh.ch IFIP Networking 2015, Toulouse, France, May 20, 2015 QoE Models for VoIP DQX and Goals Experiments and Results Conclusion
- 2. © 2015 UZH, E-model (R) Ro – Various noise sources Is – Loud speech level – Non-optimum Overall Loudness Rating (OLR) – Non-optimum Side Tone Masking Rating (STMR) Id – Delay – Echo Ie – Equipment impairment factor A – Expectation R = 0R − sI − dI − eI + A
- 3. © 2015 UZH, IQX Hypothesis IQX :QoE = α ×e−β×QoS +γ 1 degree of freedom – β: curve gradient α and γ define the min and max Mean Opinion Score (MOS) 0-1 normalized value of a variable MOS
- 4. © 2015 UZH, DQX Model Increasing Variable (IV) – The more you have the better it is Decreasing Variable (DV) – The more you have the worst it is Mixed Variable – Multiple variables affect QoE
- 5. © 2015 UZH, DQX HOWTO Formalizing QoE in 6 steps 1. Identify variables that affect QoE 2. Characterize those variables • Increasing variables (IV) • Decreasing variables (DV) 1. Select the ideal/desired/expected/agreed value of a variable 2. Considering the service specifications select the best and the worst value of each variable 3. Identify the effect of each variable’s variation • Influence factors (m) 1. Identify the importance of each variable (wk)
- 6. © 2015 UZH, DQX Model ed (x) = 4e − x x0 ÷ m ln4 3 +1QoE equation for DVs ei (x) = 4(1−e − x x0 ÷ m ln4 )+1QoE equation for IVs E(X) =1+ 4 e i∨d( ) xk( ) −1 4 k=1 N ∏ wk Generic QoE equation Importance factor Step 6 Influence factor Step 5 Expected value Step 3 Variables selection Step 1 Variables characterization Step 2 QoE QoE-related variables values Best and worst values Step 4
- 7. © 2015 UZH, DQX Model Influence Factor m Exponential functionLinear function Step function
- 8. © 2015 UZH, Goals Define and calibrate the parameters of DQX in the VoIP scenario Collect QoE-related feedback Develop a QoE measurement setup wrt – Latency – Packet loss – Jitter – Bandwidth Compare DQX with state of the art QoE models in VoIP – IQX Hypothesis – E-model
- 9. © 2015 UZH, Experiment Setup Network Emulation • Jitter • Latency • Packet loss • Bandwidth Real-Time Communications (RTC) Wide Area Network emulator (WANem)
- 10. © 2015 UZH, Experimental Calls 34 Subjects Places – IFI UZH – KS Willisau 6 hours – 541 data points 45 different Scenarios – 80% single variable – 20% mixed variables
- 11. © 2015 UZH, Evaluation Single variable scenarios – Variables • Latency • Packet Loss • Jitter • Bandwidth – m values Comparison – DQX – IQX – E-Model Mixed variables scenario
- 12. © 2015 UZH, min/max and Expected Variable Values x0 Latency – min value = 0 ms: no delay – x0 = 150 ms: codec independent, ITU-T recommendation G.114 and G.1010 – max value = 1800 ms: satellite connection Jitter – min value = 0 ms: no jitter – x0 = 100 ms: no values for Opus in literature, Cisco recommendation – max value = 1800 ms Packet Loss – min value = 0%: no packet loss – x0 = 5%: official Opus codec documentation – max value = 50% Bandwidth – min value = 0 kBit/s: no connectivity – x0 = 64 kBit/s: default bandwidth for WebRTC according to its documentation – max value = 140 kBit/s
- 13. © 2015 UZH, Evaluation: Packet Loss
- 14. © 2015 UZH, Evaluation: Latency
- 15. © 2015 UZH, Evaluation: Jitter
- 16. © 2015 UZH, Evaluation: Bandwidth (m-:4.45, m+:0.47)
- 17. © 2015 UZH, Influence Factor (m) Escalation Variable’s Value
- 18. © 2015 UZH, Influence Factor (m) Escalation - Bandwidth
- 19. © 2015 UZH, Evaluation: Mixed Variables 14 scenarios, unadjusted importance factor wk Mean Opinion Score (MOS) difference (Collected – DQX) : 0.53 Standard Deviation: 0.68
- 20. © 2015 UZH, Conclusion & Future Work Conclusion – DQX is flexible – Influence factor m is not constant – Importance factors w and further calibration of the min, max, expected values can improve the DQX results – Critical thoughts • Subjects: men between 20 and 25 • Headsets and duration of the test calls • WebRTC, Browser Interoperability Future Work – QoE measurement setup • Other variables • More tests • Different services – Videoconference – Video streaming – Further analysis of the m value and the formula for mixed variables
- 21. © 2015 UZH, Thank you! Q&A
- 22. © 2015 UZH, # Steps from min to max Values
- 23. © 2015 UZH, Collected MOS for Mixed Variables Compared to the Calculated MOS
- 24. © 2015 UZH, Used Software
Editor's Notes
- #2: General comments: -Stick to one term -include some details about W
- #3: Impossible to define the parameters offline •Ro: Expresses the basic signal-to-noise ratio, including various noise sources, such as circuit noise and room noise. •Is: impairments that exist more or less simultaneously with the voice signal, such as… •Id: impairments by too long absolute delay and potential echo effects on both talker’s and listener’s side. •Ie: Equipment caused by the respective codec used and packet-loss. •A: The advantage, or expectation factor, considers the advantage of service access. E.g., a user in a region which is hard to provide connectivity, expects a lower quality
- #7: Step 4. Considering the service specifications select the best and the worst values of the variable Step 6 is another degree of freedom to calibrate the model in a better way. You can start by setting the importance factor = 1
- #9: Mention it: Several test calls
- #10: Mention Opus
- #11: Department of Informatics
- #14: Mention that those results are the outcome of our experiments + the comparison with other models
- #15: Slow speaking people and fast speaking people
- #17: The first derivative does not exist at this point Discontinuity
- #18: The fit was done with linear approximation
- #20: Terminology! Cut not cutted :p