Mad scalability: Scaling when you are not Google
0 likes1,068 views
The document details the infrastructure scaling challenges faced by Cabify from 2011 to 2017, focusing on their transition from VPS to bare metal servers and various database solutions. It highlights performance issues, solutions implemented like changing database technology and load balancing strategies, and the importance of monitoring and system administration. The narrative showcases the evolution and lessons learned in handling increased loads and optimizing their service architecture over the years.
Mad scalability: Scaling when you are not Google
- 1. Scaling when you are not Google Abel Muiño
- 2. Abel Muino ‣ Lead Software Engineer ‣ Tweets as @amuino ‣ In another life, co-owned 1uptalent.com, played with Docker and used AWS for everything.
- 3. Disclaimer ‣ Cabify is 5 years old ‣ I joined Cabify about 1.5 years ago to work on product ‣ What you will hear today might be ‣ 70% folklore / 30% experience ‣ Only about production ‣ Not applicable to other areas (data analytics)
- 5. Cabify
- 6. 2011 2012 2013 2014 2015 2016 Completed Journeys (Axis has no legend because NDA and stuff)
- 7. Backend committers 0 5 9 14 18 2011 2012 2013 2014 2015 2016
- 8. We are hiring! (As if it wasn’t obvious from the charts)
- 11. Cabify foundations ‣ Mostly Ruby, some Go ‣ Running on VPS ‣ No sysadmins (devops?) ‣ CouchDB ‣ Redis ‣ Home-grown metrics & monitoring (limited)
- 12. Servers ‣ 3 ⨉ Host servers ‣ Horizontally scalable ‣ Most services included (sidecars) ‣ Front + Back + Queue workers ‣ 1 ⨉ Realtime server ‣ Single Point of Failure ‣ Ansible for setting them up VPS Provider LB web1 web2 web3 worker1 LBLB redis1 redis2elastic realtime osrm websock
- 13. CouchDB ‣ Used to be run in-house → Unreliable ‣ Moved to Cloudant ‣ Managed ‣ Bare metal servers ‣ Requisite for everything else: to run on the same datacenter ‣ …because the network matters Database of choice for Cabify
- 14. Pros ‣ Cheap servers ‣ Profesional DB management ‣ Still cheaper than in-house staff ‣ Scales up by either ‣ Emailing Cloudant ‣ Deploying new VPSs ‣ Datacenter lock-in ‣ Scarce visibility on load ‣ Low VPS utilization (for some services) Cons
- 15. Tl;dr: everything was fine Until it wasn’t
- 16. 2015 Road to bare metal
- 17. In 2014 we handled 7 times the load of 2013
- 18. Installed NewRelic ‣ Monitors our ruby stack ‣ Built custom adapters for API toolkit and CouchDB ‣ Golang not supported 😭 ‣ Low hanging fruit for increasing performance ‣ Hint: Always contact a Sales Rep ‣ Bye bye home-grown monitoring! 👋
- 19. VPS provider DDoSed ‣ Several times a week ‣ Cabify was unreachable ‣ VPSs where unreachable on the internal network ‣ Slow & bad support ‣ Reputation ‣ Solution: Level up!
- 20. Nobody ever got fired for choosing IBM Moved to Bare Metal @ Softlayer Same guys hosting our Cloudant cluster 👍
- 21. Mindset Control the core, minimise work for everything else
- 22. Everything must go VPS Provider web1 web2 web3 worker1realtime LBLBLB redis1 redis2elastic osrm subscriber
- 23. Load Balancer ‣ Multiple PoP (starting operations in several countries) ‣ CDN ‣ Supporting websockets ‣ … and Load Balancing ‣ Low TCO ‣ https://www.incapsula.com
- 24. Redis, ElasticSearch ‣ Same datacenter ‣ Completely managed ‣ Clustered / reliable ‣ RedisLabs ‣ Bonus: Memcached ‣ Qbox
- 25. OSRM ‣ Same datacenter ‣ Completely managed ‣ Enhanced dataset ‣ Google Maps & Places (with enterprise license) ‣ 2 / 3, good enough
- 26. Can do better? Can we manage less infra? Softlayer web1 web2 web3 worker1realtime Google subscriber Incapsula RedislabsRedislabsRedislabs qboxqboxQbox RedislabsRedislabsCloudant
- 27. Subscriber ‣ Felt like reinventing the wheel ‣ Looked for battle-tested bus / queue / broker ‣ In the same datacenter ‣ Had previous experience with RabbitMQ ‣ CloudAMQP Homebrew message bus / queue
- 28. Sidecars ‣ Every server could run Cabify ‣ All services installed ‣ Except Realtime (SPOF) ‣ Horizontal scaling ‣ Good server utilisation (bare metal servers are larger) Make each host self-sufficient
- 29. Cut own servers in 50% Served 5 times more requests Softlayer host01 host02 host03 realtime Google Incapsula RedislabsRedislabsRedislabs qboxqboxqbox RedislabsRedislabsCloudant CloudAMQPCloudAMQP
- 30. Pros ‣ Same-datacenter latencies ‣ Only care about our product ‣ Still cheaper than in-house staff ‣ Scales up by either ‣ Emailing a provider ‣ Deploying new Servers ‣ Good visibility on perf ‣ Datacenter lock-in ‣ Still no visibility on Golang perf ‣ Competing services on each server with different needs ‣ Fast & light http requests ‣ Slow & heavy queue workers ‣ Debugability Cons
- 31. Tl;dr: everything was fine Until it wasn’t
- 32. 2016, pushing to the limit
- 33. In 2015 we handled 5 times the load of 2014
- 34. In 2016 we would invade LatAm (new countries, cities, marketing…)
- 35. Bumps on the road ‣ Start seeing intermittent latency spikes on Cloudant ‣ Disable some services, get back on track ‣ Tied to peak hours ‣ We lived through these, but was stressful
- 36. Be easy on the database ‣ Removed frequent N+1 queries patterns ‣ Moved some queries to ElasticSearch ‣ Started caching more on Memcache ‣ Grew the cluster ‣ From 200ms to 100ms (average) 👏 (trying to sleep better)
- 37. RabbitMQ can’t cope ‣ We saturated the cluster CPU with moderate load ‣ Tied to us using tag-based routing ‣ Messages were delivered much later than expected ‣ Made changes to use simpler routing ‣ Is there anything simpler than RabbitMQ for simple routing? 🤔
- 38. Interlude DynDNS goes down, Cloudant uses them We lose access to our databases cluster load balancer Patched /etc/hosts with the actual ips in 30 minutes
- 39. The right tool for the job ‣ Clouchdb / Cloudant, not the best database for frequent updates ‣ Looking for alternatives to store fast-changing models ‣ RethinkDB ‣ Fast, easy to use, hosted options in same datacenter ‣ Streaming query updates Expecting growth in line with previous years
- 40. Broke RethinkDB load balancer Database stats were OK, but the LB couldn’t handle our rate Slow support, no “enterprise” option Decided to phase out RethinkDB
- 41. Wrote our first «database» Simple in-memory store, backed by Couchdb Update indexes on writes. All queries are indexed Implemented in Golang, consumed from Ruby Replaces RethinkDB, which replaced CouchDB
- 42. Cloudant latency spikes fixed! Grow the cluster for the second time in the year Load balancers hardware upgraded, problems gone Also reduced the number of connections from ruby
- 43. Relax the Sidecars ‣ Load on background workers interfering with serving http ‣ Split the servers: ‣ Front (ruby/golang http interface) ‣ Workers (ruby job queues, ruby background)
- 44. Remove RabbitMQ Replace with NSQ Nice mix of sidecar and discovery
- 45. Softlayer Multiplied own servers by 3 Served 4 times more requests Google Incapsula RedislabsRedislabsRedislabs qboxqboxqbox RedislabsRedislabsCloudant CloudAMQPCloudAMQP host01-09host01-09host01-09host01-09 rt01-02rt01-02 work01-03work01-03work01-03
- 46. Pros ‣ Despite the problems, we had top-notch support from Cloudant ‣ Easy to scale out ‣ In-process database opened doors to new features ‣ Datacenter lock-in ‣ Still no visibility on Golang perf Cons
- 47. Cabify @ 2017
- 48. In 2016 we handled 4 times the load of 2015
- 49. Hired our first full-time sysadmin!
- 50. Taking ownership Improve our infra
- 51. Own load balancers ‣ Still use Incapsula for its PoP ‣ Achieved much better load balancing ‣ 3 new dedicated servers Better control & traceability
- 52. Plans for the future
- 53. Own redis cluster ‣ Migrating from Redislabs hosted to Redislabs Enterprise ‣ hosted used virtual servers ‣ we rely heavily on redis (and memcached) ‣ 3 new dedicated servers ‣ WIP Better control & traceability
- 54. Ruby → Elixir ‣ Fun to code with ‣ Higher performance ‣ Less memory ‣ Investment, about to release first service to production
- 55. Extract from Product Dedicated teams and resources for specific components Make the core of Cabify leaner
- 56. Thanks! And sorry for the 60 slides
- 57. Questions?