Stack Exchange Infrastructure - LISA 14
- 1. The Stack Exchange Infrastructure Vroom Vroom
- 2. inet.perf.profile • SRE Generalist @ Stack Exchange • @GABeech • http://brokenhaze.com • http://stackexchange.com
- 3. A brief Overview • 560 Million Page Views a Month • 34TB of Data transfered a Month • 1665 rps (2250 peak) Across web Farm • WISC(HER)
- 4. Our First Priority is Performance Nobody likes a slow site, least of all us. When your site is slow people leave. Make your site fast, and the people will stay Good write up on moz.com: http://moz.com/blog/site-speed-are-you-fast-does- it-matter
- 5. The Performance toolkit • Mini Profiler • OpServer (https://github.com/opserver/Opserver) • Client Timings (http://teststackoverflow.com/)
- 7. OpServer
- 10. OpServer SQL
- 11. Client Timings
- 12. You can’t be fast if you are not up • Highly Redundant network • Datacenter, ISP, Edge, Core, Server, Port
- 14. Load Balencers • HAProxy • 2 Servers (Hot/Standby) • Multiple Tiers (HAProxy Processes)
- 16. SSL Termination • Terminated at LB • Feature added to HAProxy 1.5 • See: http://brokenhaze.com/blog/2014/03/25/ how-stack-exchange-gets-the-most-out-of- haproxy/
- 17. Web Servers • IIS • 9 Production (2 Test/Dev) • Dell R610’s • 32GB Memory • 2xE5-5640
- 18. Data Tier • MS SQL Server • 4 Servers • 2 Always-On Clusters • Each Cluster 1 RW, 1 RO
- 19. Caching Tier • Redis • 2 Servers • Hot / Standby configuration
- 20. Tag Engine • Our Special index of SO • Tagging is hard • Written by Marc Gravell • http://blog.marcgravell.com/2014/04/technica l-debt-case-study-tags.html
- 21. Elastic Search • 203GB Index • 3 Machines • 42M searches/day
- 22. Deployment • Git • TeamCity • Custom Powershell Scripts
- 23. So what does this get you • 52 ms homepage render time • 33 ms questions page render time
- 24. Always See our Performance • http://stackexchange.com/performance
- 25. Thank YOU! Contact: @GABeech george@stackoverflow.com Office Hours: Wednesday, November 12th (today…) 2:00pm - 3:30pm LISA Lab
Editor's Notes
- #4: Windows IIS SQL Server C# HAProxy Elastic Search Redis
- #5: Why do I bring up performance in an infra talk? simple. It drives our design decisions.
- #7: Shown to every Dev/SRE on every page Oneboxed in our chat system
- #8: Bubbles up problems
- #12: How well are we actually doing when _you_ load the page
- #13: The actual design starts now.
- #14: 4 Different providers Selected for different characteristics Router Redundancy Hot/Standby HSRP/BGP on “T2” Full BGP tables and HSRP on T1
- #15: 4B requests/month 3000 req/sec peak 10% CPU 18% peak Between 600k and 700k concurrent connections (EST, TIME_WAIT, ETC) Multiple Processes Allow for granular restarts and segregation of faults SSL Termination done on the LB Websockets: The weird connection Long lived TCP not HTTP
- #16: Request flow In, is http? yes, servers: no term https, is http
- #17: Source Port Exhaustion use 127.0.0.0/8 to resolve Server only running at ~12% cpu We don’t run full SSL everywhere yet
- #18: 185 req/s 250 peak 15% CPU usage 20% peak
- #19: (SO) 343 M Queries per day (SO) Peak of 7500 queries / second (SE) 216M Queries per day (SE) Peak 3200 queries / second CPU Use: SO 8% Peak 15% — SE 10% Peak 20%
- #20: 3.65 B operations a day Peak 60,000/s 3% cpu usage
- #21: 3 Servers, 32 GB RAM 3644 req/s 3% CPU 10% peak Replaced Full Text search in SQL Server Spins up a full copy of SO/SE Cool thing can be upgraded with 0 downtime
- #22: 2 others/ not prod Machine learning Log stash (300TB)
- #23: Team City monitors our Development Git repository Dev Auto builds (Deploy to Meta) When the build is verified Dev triggers Prod Build Copy Artifacts from Dev Build