Redis and its Scaling and Obersvability
- 2. Agenda ➔ Redis Quiz ➔ Redis Overview ➔ Setup of Redis ➔ Scaling in Redis ➔ Observability of Redis
- 3. Redis Overview Q1 - According to your understanding, what do you mean by Redis? A. It’s a NoSQL database B. It’s a caching system C. It’s a SQL database. D. Both A and B D. Both A and B
- 4. Redis Overview Q2 - What are the different type of Redis setup are possible? ● Standalone setup ● Replicated cluster setup ● Sharded cluster setup
- 5. Redis Overview Q2 - Can we use Redis as a system with the traditional SQL databases? A. True B. False A. True
- 6. Redis Overview In-Memory Database Data stored on system/server’s memory Usage of Redis Often used as Cache to improve performance Redis can also be used as full fledged database as well. Application Redis Database
- 7. Usage of Redis Client Application Redis MySQL DB 1. Looks in cache system. 3. Cache the new data Cache hit simply return the data. 2. Cache miss will look into the main persistent database.
- 8. Key Properties of Redis In-Memory Database Fast and Performant Schemaless
- 9. Types of Redis Setup Sharded Cluster Replicated Cluster Standalone Standalone Leader Follower Follower Leader Leader Leader Follower Follower Follower
- 10. Standalone Setup Redis Server Client Applications A single application or multiple applications can connect to the Redis server for writing and reading keys.
- 11. Challenges of Standalone Setup Out of Memory Data Bottleneck Failover Scalability
- 12. Replication Leader A B C D Follower A B C D Follower A B C D
- 13. Sharding Leader 1 A B Leader 2 C D Leader 3 E F
- 14. Replication + Sharding Leader 1 A B Leader 2 C D Leader 3 E F Follower 1 A B Follower 2 C D Follower 3 E F Replication of data from leader to follower Redirection of the data
- 15. Replicated Setup Redis Leader Client Applications Redis Follower Redis Follower Redis follower is replicating the data that is being written on the redis leader server Client application will write the data on the leader server of Redis All read requests will be redirected to the followers node to reduce the network noise from leader node of Redis.
- 16. Replicated Setup with Sentinel Redis Leader Redis Follower Redis Follower Redis follower is replicating the data that is being written on the redis leader server Redis Sentinel Redis sentinel keeping watch on replicated cluster, if leader will fail, it will promote the follower to the leader System Failure Redis Leader Redis Follower
- 17. Challenges of Replicated Setup Redis Leader Redis Follower Redis Follower ● Dataset can become large for leader and follower node which can crash Redis process. ● A single master can become a bottleneck for the data
- 18. Sharded Cluster Setup Leader 1 Leader 2 Leader 3 Follower 1 Follower 2 Folower 3 Leader is sharding data between the leader nodes Followers replicating data from leader nodes Redirection for data search and write Write requests can be sent on leader nodes Read requests can be sent on follower nodes Client Application Client Application
- 19. Setup Comparison Standalone Replicated Setup Sharded Setup Data Bottleneck High Medium Low Out of Memory Risk High Medium Low
- 20. Setup Comparison Standalone Replicated Setup Sharded Setup No chances Medium chances High chances Low Medium High Failover Scalability Complexity of Setup Low Medium High
- 21. Scaling Redis Setup Redis Standalone Redis Leader Redis Follower Redis Follower Leader 1 Leader 2 Leader 3 Follower 1 Follower 2 Folower 3 Replicated Setup Sharded Setup
- 22. Scaling Replicated Setup Redis Leader Redis Follower Redis Follower Redis Sentinel Redis Follower Redis Sentinel Redis Sentinel Scaling replication followers Scaling sentinel for failover and cluster
- 23. Scaling Sharded Setup Redis Leader 1 A B Redis Leader 2 C D Redis Leader 3 F G Redis Follower 1 A B Redis Follower 2 Redis Follower 3 E H C D E F G H Redis Leader 4 Redis Follower 4 H E H E Resharding
- 24. Challenges in Scaling Redis Sentinel configuration and scaling Resharding of data in cluster mode Complexity in scaling Redis
- 25. Monitoring of Redis Redis Redis Exporter Prometheus Grafana Dashbaord Prometheus pulling metrics from the redis exporter at defined interval Grafana using Prometheus as the datasource for metrics Panels and dashboards created on Grafana platform
- 26. Important Monitoring Metrics for Redis Uptime Latency Expired vs non-expired keys Size of DB CPU Memory Througput Connection Count
- 28. Redis Setup on Kubernetes Standalone Redis Pod Leader Follower Follower Leader Follower Leader Leader Follower Follower
- 29. Challenges of Redis On Kubernetes Dynamic IP update of redis pod Scaling Redis cluster on- fly Monitoring of Redis system Production ready configuration Security in terms of Kubernetes and Redis Managing persistent data
- 30. Solution of Redis on Kubernetes OT Redis Operator Host, Run and Scale Redis cluster on Kubernetes.
- 31. Features of Redis Operator Multiple Redis version supported Standalone and Cluster mode supported Automated failover and recovery is supported In built monitoring support with exporter Password and password-less support TLS support for security ipv4 ipv6 Ipv4 and Ipv6 support for Redis Detailed monitoring dashboard of Grafana On-fly upgrade without downtime
- 33. Standalone Setup of Redis
- 34. Cluster Setup of Redis
- 37. TLS Integration
- 38. Contributors Behind Redis Operator Abhishek Dubey, Sr. DevOps Consultant, Opstree Solutions Sandeep Rawat, CTO, Opstree Solutions Shubham Gupta, DevOps Intern, Opstree Solutions Mathieu Cesbron, Developer from France David Collam, Staff Solutions Engineer, Jetstack (Cert Manager) Ivyan, Software Engineer, DaoCloud
- 39. Contributors Behind Redis Operator Andrew Thompson, Software Engineer, CrowdStrike John, Software Engineer, Naver Hubert FONGARNAND, Software Engineer, Ippon Anish Chakraborty, Sr. Software Engineer, Spotify Tanuj Dwivedi, Sr. Software Engineer, Dream11 Philippe Martin, Software Engineer, RedHat
- 40. And Many More….
- 41. Thank you! Contact:- abhishek.dubey@opstree.com, Sandeep@opstree.com http://opstree.com http://buildpiper.io