Managing 50K+ Redis Databases Over 4 Public Clouds ... with a Tiny Devops Team
Download as PPTX, PDF26 likes28,745 views
Redis is an in-memory database that provides fast performance for powering lightning fast apps. It supports many data structures like strings, hashes, lists, sets and sorted sets. Redis is efficient due to its support for many data structures and commands, as well as its complexity-aware design. Redis Labs provides fully-managed cloud services for Redis and Memcached, and helps customers with challenges around scalability, high availability, performance and monitoring for large-scale Redis deployments.
Managing 50K+ Redis Databases Over 4 Public Clouds ... with a Tiny Devops Team
- 1. 1 powering lightning fast apps
- 2. 2 The newest NoSQL The fastest data store available today (served entirely from RAM) Among the top 3 databases chosen by developers Much more than a simple key/value - Strings, Hashes, Lists, Sets, Sorted Set, LUA, transactions, Bits operations Strong use cases, dynamic community, large eco-system Redis
- 3. 3 Leading the commercial Redis market Founded in 2011; GA in 02/2013 2,400+ paying customers; 52,000+ DBs; 100+ new DBs/day 2nd largest contributor to open source Redis Raised $13M - Bain/Carmel/Strategic/Angels Offices in Santa Clara and Tel-Aviv Redis Labs
- 4. 4 Redis Cloud Memcached Cloud Our offering Fully-managed cloud services. On-prem server license - soon. 4
- 5. 5 100msec = Fast apps requirements max E2E response time, under any load 50msec = average Internet latency 50msec = required app response time (includes processing & multi DB accesses) 1msec = required DB response time The only database to meet requirement=
- 7. 7 Why is Redis efficient ? Many data-structures Many cool commands (atomicity maintained) Complexity aware
- 8. 8 Real world use case: • 500+GB • 400K writes/sec • 1500 reads/sec • 37.5KB average object size Efficiency No extra work at app level 1.5Gbps 120Gbps Tones of work at app level NoSQL 6 Nodes cluster 150+ Nodes cluster
- 9. 9 Timeline Followers Caching Messaging Geo search Leaderboards Job management RT analytics Verticals & main use cases Online advertising Social Gaming Financial Services
- 10. 10 • Multi-TB in memory • ~ 300,000 reads/sec • ~ 5,000*N writes/sec N - # of followers Twitter Every Timeline (800 tweets per user) is on Redis
- 11. 11 • 20TB+ in memory • ~ 6,000,000 reads/sec • ~ 600,000 writes/sec Weibo (Chinese Twitter) • Counting • Reverse cache • Top 10 lists • Last Index • Relational list/Message Queue • Fast transactions w/ LUA
- 12. 12 Pinterest Object graph: • Per user (Sorted Set w/ timestamp as score) store the users followed (explicit+ implicit) store the user’s followers (explicit+implicit) • Per board Redis Hash for storing explicit followers Redis Set for storing explicit unfollowers
- 13. 13 Stack Overflow Three levels of cache: • Local cache (no persistence) sessions, and pending view count updates • Site cache hot question id lists, users acceptance rates.. • Global cache Inboxes, API usage quotas, …
- 14. 14 Github • Redis is used for routing info • Matching user repositories to server names
- 15. 15 Hipchat • Which users are in which room • Who is online • XMPP server balancing
- 16. 16 Youporn Most data is found in Hashes with ordered Sets used to know what data to show (1) ZinterStore on: {videos:filters:release}{videos:filters:orientation:straight} {videos:filters:categories(id)}{videos:ordering:rating} (2) Perform a ZRANGE to get the pages we want and get the list of video_ids back (3) Start pipelining to get all the videos from Hashes
- 17. 17 Snapchat • 500+ instances • 15-50TB • Running on GCE 400M messages/day
- 18. 18 Why Redis Labs ?
- 19. 19 Infinite seamless scalability True high-availability Stable top performance Zero management Users choose us because..
- 20. Dynamic Clustering Technology Zero-latency proxy Cluster manager In-Memory Node Cross-shard processor In-Memory Cluster +
- 21. 21 Challenge #1 How to serve users from the same data-center ?
- 22. 4 clouds /10 regions
- 23. 18 data-centers / 30 clusters
- 24. 24 AWS zones mapping dilemma Redis Labs User us-east-1a us-east-1c us-east-1b us-east-1c us-east-1e us-east-1d us-east-1a us-east-1e us-east-1b
- 25. 25 Eric Hammond’s post on: Matching EC2 Availability Zones Across AWS Accounts How did we solve it
- 26. 26 How did we solve it Redis Labs User
- 27. 27 Challenge #2 Which instance type shall we use for our cluster?
- 28. 28 Various instance types in the same cluster • High load scenarios • High memory usage scenarios • New generation of instances Dedicated instances As cheap as possible Cluster’s node requirements
- 29. 29 Adrian Cockcroft's Blog - Understanding and using Amazon EBS - Elastic Block Store • use large instances and get dedicated instances for free The tip
- 30. 30 What we use today C3 & R3 A4/5/6/7 n1-standard n1-highmem n1-highcpu BM+VM
- 31. 31 Challenge #3 How to mange data-persistence with high volumes of ‘writes’ and slow cloud storage ?
- 32. 32 Ephemeral vs. Persistence storage Ephemeral EBS/Cloud Drive/Persistent Disk/SAN Network attached Persistent Slow Direct attached Ephemeral “Fast”
- 33. 33 Adrian’ s Blog use the larger EBSes if you want speed Google (GCP) “Larger volumes can achieve higher I/O levels than smaller volumes” The tips
- 34. 34 We use large volumes (1TB+) We use both ephemeral and persistent storage We improved/tuned/optimized the Redis persistent storage interface If replication is enabled, slave writes to disk We don’t use PIOPS What we do
- 35. 35 Why not PIOPS
- 36. 36 Challenge #4 How to monitor 50K+ databases, 30+ clusters and hundreds of nodes ?
- 37. 37 Zabbix (not Nagios) - per node metrics Limbic (home made) - databases’ metrics • 50K (databases) x 100+(metrics) x 10K+(time resolutions) • Based on Python, RRD, Redis Redis adminUI – cluster configuration Monitoring
- 39. 39 Team /Method/Spirit Tiny devops team Core dev. team knows ops (very well) Baby steps, especially in production The practical approach always wins Review your plans every 3 months
- 40. 40 We are hiring !
- 41. 41 Thank You
- 42. 42 Why is Redis efficient ? Many data-structures Many cool commands (atomicity maintained) Complexity aware
- 43. 43 Think data-structure • Strings • Hashes • Lists • Sets Sorted Sets • HyperLogLogs
- 44. 44 Cool commands • SET if it doesn’t exist – O(1) • Blocking POP (with timeout) – O(1) • (blocking) POP from one list, PUSH to another – O(1) • Get/Set string ranges (and bit operation) – O(N) • Union/Intersect/Ranges of SETs – O(N)+O(Mxlog(M)) • Pub/Sub – O(1)/O(M)/O(M+N) • LUA / Transactions / Pipelining
Editor's Notes
- #3: Choozen
- #4: 20 13 Cont ributer
- #7: At bottom An at the top is Redis
- #10: Let’s run through it quickly
- #20: What are the benefits of using us You performance can vary quite a bit
- #21: Maa ner Multi-tenant technology As many as you want Without further ado let’s go to the demo
- #23: Insert Version Number Here
- #24: Insert Version Number Here
- #26: 20 13 Cont ributer
- #27: 20 13 Cont ributer
- #29: 20 13 Cont ributer
- #30: 20 13 Cont ributer
- #31: 20 13 Cont ributer
- #33: 20 13 Cont ributer
- #34: 20 13 Cont ributer
- #35: 20 13 Cont ributer
- #36: 20 13 Cont ributer
- #38: 20 13 Cont ributer
- #40: 20 13 Cont ributer
- #44: Provides you with