Redis sentinelinternals deview
30 likes6,822 views
1. Redis Sentinel provides high availability for Redis databases by monitoring Redis servers, detecting failures, and initiating failovers to slave servers. 2. When a failure is detected, Sentinel will promote a slave to become the new master, redirect clients to the new master, and reconfigure other slaves to connect to the new master. 3. While Sentinel provides basic high availability, it has some limitations such as not being able to promote a slave if the original master also becomes a slave, and not being able to handle Redis servers that are loading data during startup.
![Checking Liveness
servers = [ ('localhost', 6379),
('localhost', 6380),
('localhost', 6381) ]
conns = []
def check():
for conn in conns:
print conn.ping()](https://image.slidesharecdn.com/redissentinelinternalsdeview-131015080519-phpapp02/85/Redis-sentinelinternals-deview-9-320.jpg)
![Checking it works
def check():
for conn in conns:
i = conn.info()
print i['run_id'],
i['used_memory_human'],
i['total_commands_processed']](https://image.slidesharecdn.com/redissentinelinternalsdeview-131015080519-phpapp02/85/Redis-sentinelinternals-deview-11-320.jpg)
![Promoting
Send Slaveof no one to new master
Send Slaveof [new master ip] [addr] to
Other redis
Notify new master to clietns
+switch-master
Set DEMOTE mark to old-Master](https://image.slidesharecdn.com/redissentinelinternalsdeview-131015080519-phpapp02/85/Redis-sentinelinternals-deview-30-320.jpg)
Redis sentinelinternals deview
- 2. Redis.io
- 9. Checking Liveness servers = [ ('localhost', 6379), ('localhost', 6380), ('localhost', 6381) ] conns = [] def check(): for conn in conns: print conn.ping()
- 11. Checking it works def check(): for conn in conns: i = conn.info() print i['run_id'], i['used_memory_human'], i['total_commands_processed']
- 12. Is it Reliable?
- 14. Split Network.
- 15. Server is busy.
- 16. Monitoring is not the one we really want.
- 17. HA High Availity
- 18. What do you do? When your servers failed
- 19. 1. Recognize Redis is down.
- 20. 2. choose new master candidate
- 21. 3. Promote slave to Master.
- 22. 4. make client connect to new master
- 24. 1. Check Redis Liveness
- 25. 1. Check Redis Liveness -Subjective Down -Objective Down
- 26. 2. Choose good slave.
- 27. Good Slave NOT SDOWN, ODOWN, DISCONNECTED NOT DEMOTE Ping reply > info_validity_time Slave_priority != 0 Info reply > info_validity_time
- 28. Choose new master Sort Slave Priority Runid
- 29. 3. Promote it
- 30. Promoting Send Slaveof no one to new master Send Slaveof [new master ip] [addr] to Other redis Notify new master to clietns +switch-master Set DEMOTE mark to old-Master
- 34. Choose Slave A as New Master Master Slave A Slave B Sentinel Pub/Sub Client Client
- 35. Send to Slave B Slaveof Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
- 36. Send “Slaveof no one” to Slave A Master Slave A Slave B Sentinel Pub/Sub Client Client
- 37. Send “Slaveof Slave A” to Slave B Master Master Slave B Sentinel Pub/Sub Client Client
- 40. Sentinel Failover State Machine State 내용 NONE WAIT_START SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION RECONF_SLAVES DETECT_END Next State SELECT_SLAVE SEND_SLAVEOF WAIT_PROMOTION
- 41. Starting Point Of Sentinel • sentinelTimer in sentinel.c –Called every 100ms –sentinelHandleRedisInstance
- 42. SentinelRedisInstance • master –If it is master, it is null. • slaves • sentinels
- 43. sentinelHandleDictOfRedisInstances while((de = dictNext(di)) != NULL) { sentinelRedisInstance *ri = dictGetVal(de); sentinelHandleRedisInstance(ri); if (ri->flags & SRI_MASTER) { sentinelHandleDictOfRedisInstances(ri->slaves); sentinelHandleDictOfRedisInstances(ri->sentinels); …… } }
- 44. sentinelHandleRedisInstance • Reconnect to Instances • Ping • Asking Master State to other Sentinel • Check SDOWN • Check ODOWN
- 45. How to check Subjective Down • When Sentinel get Ping Reply –Set last_avail_time as current • Check –e = current –last_avail_time –e > last_avail_time •Subjective Down.
- 46. How to check Objective Down • Ask to other sentinels is it down. –sentinelAskMasterStateToOtherSentinels • If other sentinels reply it is down. –Count them, if it is bigger than Quorum. –It is objective Down.
- 47. How to find Redis
- 50. +redirect-master
- 51. How to find Sentinel
- 52. Find Sentinel Subscribe SENTINEL_HELLO_CHANNEL In Master Publish Sentinel Information In sentinelPingInstance Add New Sentinel In sentinelReceiveHelloMessages
- 54. Is Sentinel Really Good?
- 55. Not Mature #1
- 56. Sentinel Conf port 26379 sentinel monitor mymaster 127.0.0.1 6379 2 sentinel down-after-milliseconds mymaster 2000 sentinel can-failover mymaster yes sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 900000
- 57. Quorum Count sentinel monitor mymaster 127.0.0.1 6379 2 We should check this count.
- 58. Sentinel Conf sentinel down-after-milliseconds mymaster 2000 If this value is Too small, It can cause some trouble. -sdown/+sdown loop
- 59. Compare with Zookeeper Conf tickTime=2000 dataDir=/var/zookeeper clientPort=2181 initLimit=5 syncLimit=2 server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
- 60. Not Mature #2
- 68. Case #2 Master Slave Sentinel Sentinel can’t promote Slave to Master
- 69. Why?
- 70. Sentinel Mechanism. When master is changed to Slave, Reconf with new Master.
- 71. Sentinel Mechanism. When master is changed to Slave, Reset information with new Master. But, if new master failed, sentinel will not get any information from it.
- 72. Sentinel Mechanism. if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE && ri->slave_master_host) { sentinelEvent(REDIS_WARNING,"+redirect-to-master",ri, "%s %s %d %s %d", ri->name, ri->addr->ip, ri->addr->port, ri->slave_master_host, ri->slave_master_port); sentinelResetMasterAndChangeAddress(ri,ri->slave_master_host, ri->slave_master_port); return; /* Don't process anything after this event. */ }
- 73. Not Mature #3
- 74. If redis is down, when redis is loading data(RDB, AOF), sentinel can’t register redis.
- 75. Conclusion
- 76. Even though, Sentinel is not mature. It is useful.
- 78. 1. Subscribe all Sentinels And ignore duplicated message
- 79. 2. Sentinel has explicit failover command sentinel mymaster failover
- 80. But can’t select new master
- 83. Sentinels can’t share their status
- 84. Deploy conf to all sentinels with Script
- 85. And restart all sentinels.
- 86. 4. If you want not to promote specific server to master
- 87. Set Slave-Priority = 0
- 88. Thank you!
- 89. Just One more thing
- 92. Agent Pub/Sub Shard Redis Master Restart VRRP Twemproxy HA Proxy Redis Slave Redis Slave Twemproxy HA Proxy Twemproxy Shard Sentinel Redis Master Redis Slave Redis Slave