Kranky Geek 2015 - Decisions & Considerations in building your WebRTC App
2 likes763 views
The document outlines Rob Brazier's presentation on operational lessons learned while managing RTC at Twilio, emphasizing the importance of communication channels and the complexities of WebRTC. It discusses testing strategies, how to handle failures, and the significance of maintaining operational excellence through cultural practices. Additionally, it highlights various methods of monitoring service performance and the necessity for a resilient infrastructure.
Kranky Geek 2015 - Decisions & Considerations in building your WebRTC App
- 1. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER RUNNING RTC Operational lessons learned at Twilio.
- 2. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TWILIO?
- 3. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Communication Channels TWILIO COMMUNICATIONS PLATFORM VOICE SMS MMS WEB MOBILE Your Business Applications IP MESSAGING VIDEO
- 4. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER 750,000Developers & Businesses 50BILLIONAPI Interactions a year OPERATING AT SCALE
- 5. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TWILIO AND WEBRTC?
- 6. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 7. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER 2011 2012 2013 2014 Present TWILIO CLIENT 1.1: WEBRTC TWILIO CLIENT: WEBRTC CALL VOLUME WEBRTC RELEASED IN CHROME 23 TWILIO CLIENT 1.0: FLASH
- 8. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER 2011 2012 2013 2014 Present TWILIO CLIENT 1.1: WEBRTC TWILIO CLIENT 1.2: GLOBAL LOW LATENCY TWILIO CLIENT: WEBRTC CALL VOLUME WEBRTC RELEASED IN CHROME 23 WEBRTC RELEASED IN FIREFOX 22 TWILIO CLIENT 1.0: FLASH
- 9. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER 2011 2012 2013 2014 Present TWILIO CLIENT 1.1: WEBRTC TWILIO CLIENT 1.2: GLOBAL LOW LATENCY TWILIO CLIENT: WEBRTC CALL VOLUME WEBRTC RELEASED IN CHROME 23 WEBRTC RELEASED IN FIREFOX 22 TWILIO CLIENT 1.0: FLASH
- 10. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER COMMON USE CASES
- 11. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER COMMON USE CASES CRM and SaaS application integration Call center / customer service Sharing economy Education / expertise on-‐demand Recruiting Anonymous calling
- 12. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER A HANDFUL OF OUR WEBRTC CUSTOMERS
- 13. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER WEBRTC COMPLEXITY DOES NOT INCREASE LINEARLY.
- 14. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 15. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Signaling Server
- 16. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Signaling Server
- 17. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Signaling Server
- 18. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Signaling Server TURN Server
- 19. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TURN Server Signaling Server Media Server Conference State
- 20. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TURN Server Signaling Server Media Server SIP Gateway Conference State
- 21. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TURN Server Signaling Server SIP Gateway Conference State Media Server
- 22. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TURN Server Signaling Server SIP Gateway Conference State PSTN GatewayMedia Server Carrier
- 23. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TS SC PM CDC1 DC5
- 24. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER DC3 TS SC PM C DC7 DC6DC1 DC5 DC2 DC4
- 25. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER SOMETHING WILL BREAK.
- 26. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 27. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Stuff you can’t control Random host failure Network degradation Spurious network traffic Stuff you can control Resource management Operational mishaps Bugs / crashes
- 28. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 29. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Tens of failures / week. 100% up-‐time two quarters running. 99.95% up-‐time for the year.
- 30. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER TEST CONSTANTLY.
- 31. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTS. CONTINUALLY TEST THE PRODUCT THE SAME WAY THE CUSTOMER WILL.
- 32. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node-‐webrtc signaling media
- 33. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node-‐webrtc signaling media Original approach: -‐Test app built using node-‐ webrtc. -‐Acted like a full-‐fledged browser client. -‐Ran a suite of tests at high frequency (some tests multiple times / min, other tests every few minutes).
- 34. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node-‐webrtc signaling media
- 35. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node-‐webrtc signaling media Hard to maintain: -‐Difficult to keep up-‐to-‐date with WebRTC changes. -‐Build challenges (CentOS 6 on AWS). -‐(node-‐webrtc has since removed MediaStream API)
- 36. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node signaling media robocallers signaling
- 37. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING node signaling media Revised approach: -‐Frequent testing for service monitoring through simple node websocket client. -‐Automated browser regression testing and “thick” scenario testing with robocallers. robocallers signaling
- 38. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Rollbar
- 39. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING
- 40. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER END-TO-END TESTING Manual testing tools: -‐Automated tests won’t catch everything. -‐Simple, reliably manual testing tools for rapidly running standard scenarios are invaluable.
- 41. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER End-‐to-‐end tests Anomaly detection Host monitoring (Nagios) Manual tools
- 42. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER End-‐to-‐end tests Anomaly detection Host monitoring (Nagios) Manual tools Layered monitoring -‐Event aggregated in Rollbar (and assorted other tools). -‐Alerts delivered through PagerDuty.
- 43. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER PLAN FOR FAILURE, PLAN FOR RECOVERY.
- 44. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Load Balancer A Load Balancer B Gateway A Gateway B Gateway C DNS distribution wss://www.mywebrtcservice.com Registrar
- 45. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar
- 46. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar
- 47. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar
- 48. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER x Many Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar
- 49. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER x 10,000,000 Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar
- 50. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER x 10,000,000 Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar Rate limit here to protect downstream services
- 51. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER x 10,000,000 Load Balancer A Load Balancer B Gateway A Gateway B Gateway C Client Registrar Be careful about locking here
- 52. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER EXPECT LOUSY NETWORKS.
- 53. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER Network Link Conditioner
- 54. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER https://github.com/vadymy/networkThrottler
- 55. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 56. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 57. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER
- 58. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER .GETSTATS() IS YOUR FRIEND.
- 59. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER pc.getStats() AMAZON KINESIS MONITORING REDSHIFT
- 60. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER SOME OF THE STATS WE CAPTURE…
- 61. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER audioInputLevel audioOutputLevel browser jitter rtt received_packets sent_packets received_packets_lost SOME OF THE STATS WE CAPTURE…
- 62. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER SOME OF THE PATTERNS WE LOOK FOR…
- 63. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER High percentage of packets lost Excessive jitter or rtt Calls > 5 seconds with no packets sent but > 0 audio-‐ input level (or the inverse; evidence of one-‐way audio) …look for patterns by Twilio account, endpoint identity, browser version, etc. SOME OF THE PATTERNS WE LOOK FOR…
- 64. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER OPERATIONAL EXCELLENCE IS A CULTURE ISSUE.
- 65. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER SOME CULTURAL TOOLS
- 66. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER SOME CULTURAL TOOLS Write the SLA first Build it, operate it Five-‐Whys Analysis Fire Drills Runbooks Post-‐Mortems Time to Recovery Reports
- 67. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER COMMUNICATIONS TOOLS ARE ONLY AS GOOD AS THEIR AVAILABILITY.
- 68. © 2009-2015 TWILIO, INC. ALL RIGHTS RESERVEDRBRAZIER@TWILIO.COM PRODUCT MANAGER, SDKSROB BRAZIER ROB BRAZIER RBRAZIER@TWILIO.COM