Handling Redis failover with ZooKeeper
- 1. Handling Redis failover with ZooKeeper Ryan LeCompte BaRuCo Lightning Talk September 2012
- 2. Me • Rubyist at Spreecast in San Francisco • redis_failover, method_locator, synchronizable • http://github.com/ryanlecompte • lecompte@gmail.com • @ryanlecompte
- 4. Master Dies :-(
- 5. What happens to your data and connected clients? • Resque Jobs? • Sidekiq Jobs? • Cached pages? • Cached stats? • Rubygems?
- 6. Scramble to manually reconfigure slave as master
- 7. Update all Redis clients to talk to the new master
- 8. Restart production services so they can see the new config
- 9. What do we really want to do? NOTHING
- 10. Failover should be safe and automatic
- 12. Open source Apache Project (originally from Yahoo)
- 13. Provides a set of simple primitives to help you build powerful distributed applications
- 14. Automatically handles network partitions & quorum management
- 16. Used by Yahoo, Netflix, Rackspace, Hadoop, Neo4j, etc
- 17. How can I use ZooKeeper in my architecture?
- 18. Distributed Configuration (update thread pool sizes & other settings without requiring a server restart)
- 19. Distributed Locking (only one process should be able to do something at a time)
- 20. Priority Queues
- 21. Leader Election, Group Membership & more
- 22. Practical application: Using ZooKeeper to build automatic failover for Redis
- 24. Provides a smart “failover- aware” client that wraps existing Ruby client for Redis
- 25. Clients automatically connect to ZooKeeper and register for watch notifications
- 26. Node Managers monitor Redis servers for availability and automatically inform clients of failover via ZooKeeper
- 27. Architecture
- 28. RedisFailover::Client Example # create new client with list of ZooKeeper servers client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost: 2182,localhost:2183') # write operations automatically routed to master client.set(‘foo’, ‘bar’) # read operations automatically routed to a slave client.smembers(‘data’)
- 29. Companies using redis_failover • Papertrail • Backupify • MediaPiston • IDT Corporation • others
- 30. Consider incorporating ZooKeeper into your overall architecture. Utilize the primitives it gives you for solving your next distributed programming challenge.
- 31. Don’t let your Redis servers go unmanaged in production. Consider using redis_failover to sleep better at night.
- 32. Thanks for listening! http://github.com/ryanlecompte/redis_failover Questions? Ask me at the beach!