The Effect of Router Buffer Size on Subjective Gaming Quality Estimators based on Delay and Jitter
2 likes1,201 views
The document analyzes the impact of router buffer size on gaming quality, specifically addressing performance issues like delay and jitter faced in real-time services such as online gaming. It evaluates various models and theories related to buffering and its effects on subjective quality measurements, citing empirical evidence and test results. Key findings indicate that smaller buffers can lead to better gaming performance in certain conditions, particularly regarding delay and jitter management.
The Effect of Router Buffer Size on Subjective Gaming Quality Estimators based on Delay and Jitter
- 2. THE EFFECT OF ROUTER BUFFER SIZE ON SUBJECTIVE GAMING QUALITY ESTIMATORS BASED ON DELAY AND JITTER GTC Communication Technologies Group Jose Saldana Julián Fernández-Navajas José Ruiz-Mas Eduardo Viruete Navarro Luis Casadesus University of Zaragoza, Spain
- 3. Index - I. Introduction II. Related Works III. Tests and Results IV. Conclusions
- 4. Index - I. Introduction II. Related Works III. Tests and Results IV. Conclusions
- 5. Introduction - The Internet was not designed for real-time services. - First deployed ones: e-mail, file transfer CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 6. Introduction - But real-time services are being widely used: VoIP, video conference, online gaming CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 7. Introduction - Problem: Using a best effort network for a real-time service. - Users demand a quality similar to the one they are used to. - Research: Find the relationship between network impairments and perceived quality. - Delay, packet loss, bandwidth, jitter. CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 8. Introduction - E-Model: ITU’s solution that estimates perceived quality of voice, as a function of delay, packet loss, codec, etc. - Battery of surveys in order to obtain a MOS (Mean Opinion Score) model CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 9. FPS online games - Very stringent Real-time requirements: - Interactivity (video) - Players: Difficult to satisfy CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 10. FPS online games - Traffic characteristics - - UDP Small packets (100 bytes maximum) High frequency (25 to 85 pps) A universal MOS does not exist - Some games are more sensitive to delay, or packet loss, or jitter, etc. CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 11. Index - I. Introduction II. Related Works III. Tests and Results IV. Conclusions
- 12. II. Related Works - Buffer sizing problem - Measuring subjective quality for online games
- 13. II. Related Works - Buffer sizing problem - Measuring subjective quality for online games
- 14. Buffer sizing problem - Players mainly use access networks - We are considering low-end routers, i.e. the ones we can find in access networks - Drop-tail FIFO tiny buffers, of some tens of kilobytes CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 15. Buffer sizing problem - Different proposals: - Rule of the thumb: C x RTT - Maximize buffer occupancy Too much delay - Stanford model: C x RTT / sqrt(N) - Tiny buffer: 20 to 50 packets CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 16. Buffer sizing trade-off Bigger delays Bigger Buffer Bigger jitter Smaller delays Smaller Buffer Smaller packet loss CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 Smaller jitter Bigger packet loss
- 17. II. Related Works - Buffer sizing problem - Measuring subjective quality for online games
- 18. Subjective quality for FPS - MOS as a function of SRT (linear dependence) C. Schaefer, T. Enderes, H. Ritter, M. Zitterbart. “Subjective quality assessment for multiplayer real-time games”. In Proc 1st workshop on Network and system support for games (NetGames '02). ACM, New York, NY, USA, 74-78. 2002. - Influence of delay and packet loss (not developing a MOS) S. Zander, G. Armitage, “Empirically Measuring the QoS Sensitivity of Interactive Online Game Players”. In Proc. Australian Telecommunications Networks & Applications Conference (ATNAC 2004), Sydney, Australia, Dec. 2004. - Packet loss: different behavior: - Halo: Does not work with 4% loss Quake III: Works with 35% loss CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 19. Subjective quality for FPS - Only delay and jitter, but separately M. Dick, O. Wellnitz, L. Wolf. “Analysis of factors affecting players' performance and perception in multiplayer games”. In Proc. 4th ACM SIGCOMM workshop on Network and system support for games (NetGames '05). ACM, New York, NY, USA, 1-7, 2005. - First developed MOS, adapted from E-model A. F. Wattimena, R. E. Kooij, J. M. van Vugt, O. K. Ahmed, “Predicting the perceived quality of a first person shooter: the Quake IV G-model”. In Proc. 5th SIGCOMM workshop Network and system support for games (NetGames '06), ACM, New York, NY, USA, 2006. - Only delay and jitter were considered Game: Quake IV G-model: We will use it to study quality CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 20. Index - I. Introduction II. Related Works III. Tests and Results IV. Conclusions
- 21. Previous results with VoIP - E-Model uses delay and packet loss. - Jitter is not considered, as a dejitter buffer is used CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 22. Previous results for VoIP - MOS presents a monotonically decreasing behavior with background traffic E-Model MOS 5 4.5 4 MOS 3.5 3 2.5 1 call 5 calls 10 calls 15 calls 20 calls 2 1.5 1 400 450 500 550 600 650 700 750 800 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 850 900 950 1000
- 23. Scenarios of interest - A number of players connected to the server sharing the same router buffer Internet . . . Router Game & background traffic Users CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 Game Server
- 24. Test methodology - The same scenario as in VoIP - FPS game with a MOS: Quake IV - Traffic traces available from CAIA project - Only client-to-server (the most restrictive one) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 25. Test methodology - Inter-packet time and packet size histograms - 40.7 kbps / user 40 50 60 70 80 90 100 110 bytes 0 10 20 30 40 50 60 70 ms 79.5 bytes avg CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 64 pps
- 26. Test methodology - 20 players sharing an Internet connection - Buffer: Drop-tail, fixed kB size Bandwidth Buffer size 2 Mbps 3 Mbps 10 kB 20 kB 50 kB 100 kB CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 27. Test methodology - Background traffic sharing the connection - - 50% packets 40 bytes 10% packets 576 bytes 40% packets 1,500 bytes Network RTT (avg 30 ms) added offline buffer Internet . . . Router Game & background traffic Users CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 Game Server
- 28. Test results: delay RTT, 2 Mbps 450 10 kB 20 kB 400 50 kB 100 kB 350 300 ms 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 29. Test results: delay RTT, 2 Mbps 450 10 kB 20 kB 400 50 kB 100 kB 350 300 ms 250 200 Bandwidth limit 40.7 x 20 = 814 kbps 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 30. Test results: delay RTT, 2 Mbps 450 10 kB 20 kB 400 50 kB Unacceptable delay 100 kB 350 300 ms 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 31. Test results: delay RTT, 3 Mbps 450 10 kB 20 kB 400 50 kB 100 kB 350 300 ms 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 32. Test results: delay RTT, 3 Mbps 450 10 kB 20 kB 400 50 kB 100 kB 350 300 ms 250 200 Bandwidth limit 40.7 x 20 = 814 kbps 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 33. Test results: delay RTT, 3 Mbps 450 10 kB 20 kB 400 50 kB Unacceptable delay 100 kB 350 300 ms 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 34. Test results: jitter Jitter (delay standard deviation), 2 Mbps 110 10kB 100 20 kB 50 kB 90 100kB 80 ms 70 60 50 40 30 20 10 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 35. Test results: jitter Jitter (delay standard deviation), 2 Mbps 110 10kB 100 20 kB 50 kB 90 Above the bandwidth limit, the jitter decreases, as the buffer is always full 100kB 80 ms 70 60 50 Bandwidth limit 40.7 x 20 = 814 kbps 40 30 20 10 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 36. Test results: jitter Jitter (delay standard deviation), 3 Mbps 110 10kB 100 20 kB 50 kB 90 100kB 80 ms 70 60 50 40 30 20 10 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 37. Test results: jitter Jitter (delay standard deviation), 3 Mbps 110 10kB 100 20 kB The peak is reduced when bandwidth grows 50 kB 90 100kB 80 ms 70 60 50 40 30 20 10 0 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 38. Test results: packet loss Packet Loss, buffer 10kB, 2 Mbps 45% game (79 bytes avg) 40 bytes 40% 576 bytes 1500 bytes 35% 30% 25% 20% 15% 10% 5% 0% 0 200 400 600 800 1000 1200 1400 1600 1800 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 2000 2200 2400 2600 2800 3000
- 39. Test results: packet loss Packet Loss, buffer 10kB, 2 Mbps 45% game (79 bytes avg) 40 bytes 40% 576 bytes 1500 bytes 35% 30% Small packets have a clear advantage in a drop-tail size limited buffer 25% 20% 15% 10% 5% 0% 0 200 400 600 800 1000 1200 1400 1600 1800 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 2000 2200 2400 2600 2800 3000
- 40. Test results: packet loss Packet Loss, buffer 10kB, 3 Mbps 45% game (79 bytes avg) 40 bytes 40% 576 bytes 1500 bytes 35% 30% 25% 20% 15% 10% 5% 0% 0 200 400 600 800 1000 1200 1400 1600 1800 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012 2000 2200 2400 2600 2800 3000
- 41. Test results: MOS - G-Model MOS formula: x = 0.104*ping_average + jitter_average MOS = -0.00000587 x 3 + 0.00139 x2- 0.114 x + 4.37 - Packet loss is not considered, unless it is above 35% CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 42. Test results: MOS G-Model MOS, 2 Mbps 5 4.5 4 MOS 3.5 3 2.5 2 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 43. Test results: MOS G-Model MOS, 2 Mbps 5 4.5 Good quality 4 MOS 3.5 Medium quality 3 2.5 2 Bad quality 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 44. Test results: MOS G-Model MOS, 2 Mbps 5 Delay and jitter grow 4.5 Delay grows, but jitter decreases Not monotonically decreasing 4 MOS 3.5 3 2.5 2 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 45. Test results: MOS G-Model MOS, 2 Mbps 5 Small buffers present the best behavior 4.5 4 MOS 3.5 3 2.5 2 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 46. Test results: MOS G-Model MOS, 3 Mbps 5 4.5 4 MOS 3.5 3 2.5 2 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 47. Test results: MOS G-Model MOS, 3 Mbps 5 4.5 Good quality 4 MOS 3.5 Medium quality 3 2.5 2 Bad quality 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 48. Test results: MOS G-Model MOS, 3 Mbps 5 Acceptable MOS values above bandwidth limit 4.5 4 MOS 3.5 3 2.5 2 10 kB 20 kB 1.5 50 kB 100 kB 1 0 200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000 background traffic (kbps) CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 49. Index - I. Introduction II. Related Works III. Tests and Results IV. Conclusions
- 50. Conclusions - Access networks with low-end routers Importance of the buffer size Small buffers are better for real-time apps Buffer implementation can penalyze big packets We cannot separately study each network impairment Need for subjective quality estimators to calculate MOS If delay and jitter are the considered impairments, the jitter peak produces a MOS valley CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 53. Extra Slides
- 54. Online games: genres Real-time strategy Sports MMORPG FPS CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 55. FPS online games - Delay: Very important CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012
- 56. CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012