Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
0 likes828 views
This document outlines Rakuten's migration from Cassandra to ScyllaDB in order to eliminate volatile latencies and improve performance across their catalog platform. The migration resulted in significant performance enhancements, including reduced latency and improved feed publishing times, while also achieving hardware cost savings. Lessons learned during the process highlight the complexities and challenges involved in transitioning to ScyllaDB, including careful planning and team training.
Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration
- 1. Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration Hitesh Shah Sr. Engineering Manager Rakuten Catalog Platform
- 2. 2 Defining Observability for Your Cloud DBaaS April 20, 2022, 10 AM PT | 1 PM ET | 6 PM GMT UPCOMING WEBINAR
- 4. Eliminating Volatile Latencies Inside Rakuten’s NoSQL Migration Hitesh Shah Sr. Engineering Manager Rakuten Catalog Platform
- 5. 5 + For distributed, data-intensive apps that require high performance and low latency + 400+ users worldwide + Results + Comcast: Reduced P99 latencies by 95% + FireEye: 1500% improvement in throughput + Discord: Reduced C* nodes from ~140 to 6 + iFood: 9X cost reduction vs. DynamoDB + Open Source, Enterprise and Cloud options + Fully compatible with Apache Cassandra and Amazon DynamoDB About ScyllaDB 1ms <1ms 10ms 1M 10M ScyllaDB Universe of 400+ Users
- 6. 400+ Companies Use ScyllaDB Seamless experiences across content + devices Fast computation of flight pricing Corporate fleet management Real-time analytics 2,000,000 SKU -commerce management Real-time location tracking for friends/family Video recommendation management IoT for industrial machines Synchronize browser properties for millions Threat intelligence service using JanusGraph Real time fraud detection across 6M transactions/day Uber scale, mission critical chat & messaging app 6 Network security threat detection Power ~50M X1 DVRs with billions of reqs/day Precision healthcare via Edison AI Inventory hub for retail operations Property listings and updates Unified ML feature store across the business Cryptocurrency exchange app Geography-based recommendations Distributed storage for distributed ledger tech Global operations- Avon, Body Shop + more Predictable performance for on sale surges GPS-based exercise tracking
- 7. Biography slide 7 Hitesh Shah Sr. Engineering Manager ■ Passionate Engineer - Predominantly worked with bay area tech ■ Focused on development as well as operations ■ Interests - Distributed systems, Dynamically scalable platforms ■ Foodie, hiking, movie buff, reading Company Logo
- 8. Agenda ■ Overview of Rakuten Catalog Platform ■ Generic Catalog Platform on NoSql • Cassandra as a logical choice ■ Challenges with Cassandra ■ Why ScyllaDB ■ Migration Approach ■ Post Migration ■ Lessons Learned ■ ScyllaDB concerns ■ Let’s stay in touch! 8
- 10. Rakuten Catalog Platform 10 rakuten.com rakuten.co.jp
- 12. + Total Catalog items: 1B+ + Number of Items processed daily: 250M+ + Read QPS: 10k-15k per Node + Write QPS: 3k-5k per Node Data Processing Volumes 12
- 15. Product Catalog Key Features 15 + Near real-time price updates + Near real-time availability updates + Discovering purchasing trends — analytics + Discovering shopper preferences — analytics + Large volume of data + Fast growing catalog + with 100s of millions of catalog items + Catalog Enrichment Support + Low latency requirements
- 16. Cassandra as a first class NoSql DB 16 + Highly scalable - naturally horizontally scalable + Effective with huge amount of data + Fast writes (append-only) without impacting reads + Fast reads — distributed + Column-family store + appeals to enrichments of specific elements + Speed of deployment + Analytics with large data processing
- 18. + Inconsistent Performance; Volatile Latencies + JVM Issues; OOM; long GC Pauses + Single Slow node can impact cluster + Requires tons of manual operations + Horizontal scalability at stiff cost Issues with Cassandra 18
- 19. + 5x-6x Performance Improvements + Improved TCO; Better ROI + Consistent Latencies + Seastar framework + Much better parallelism + Drop-in Replacement Why ScyllaDB 19
- 20. + Enterprise Support + Private slack channel; SLAs + ScyllaDB custom shard-aware driver for Java + Incremental Compaction Strategy + Workload Prioritization + ScyllaDB Manager Why ScyllaDB Enterprise 20
- 22. Framework for Migration ScyllaDB Evaluation: + Candidate Use Cases + Data Ingestion + Serving customers + Proof of Concept + Running pipeline against ScyllaDB + Internal Benchmarking + Multiple iterations with various configurations + Tracking and evaluating ScyllaDB performance + Evaluating Shard-aware custom driver Evaluation Methodology 22
- 24. + ScyllaDB Engagement + Making a final decision + Finalizing production topology + Planning live Database migration + Training Team with ScyllaDB Making a Decision 24
- 25. + Functional Validation/Testing + Commissioning production cluster + Backup of c* cluster + Restoring Data in ScyllaDB cluster + Making ScyllaDB cluster ready + Replaying live production traffic/transactions + Switching to ScyllaDB + Post switchover production issues and resolutions + Decommissioning old c* cluster Production Migration 25
- 27. + Average Feed Ingestion - 30-35% faster + Improved SLAs + 2.5x-5x improvements in feed publishing times + Quicker turnaround for catalog changes + 21 Nodes in cassandra to 6 Nodes in ScyllaDB + Hardware $$$ savings KPI Improvements 27
- 28. + Lengthy ScyllaDB evaluation cycle + Complex Technology + ScyllaDB Ramp Up curve is non-trivial + ScyllaDB sizing is the key and is misunderstood + Finalizing the cluster topology/server configuration was the key + Migrating Live production database is challenging + detailed planning + tight execution + customer communication Lessons Learned 28
- 29. + Lack of documentation + ScyllaDB not bug free, of course + Operational and Setup complexities + Monitoring ScyllaDB (i.e. internal metrics) is costly + ScyllaDB Monitoring Stack is free but..! + Hardware resource intensive ScyllaDB Concerns 29
- 30. Sizing Poll
- 31. United States 2445 Faber St, Suite #200 Palo Alto, CA USA 94303 Israel Maskit 4 Herzliya, Israel 4673304 www.scylladb.com @scylladb Thank You! Q&A