Netflix's Big Leap from Oracle to Cassandra
Download as PPTX, PDF7 likes2,582 views
This document summarizes Netflix's migration from Oracle to Cassandra. It discusses how Netflix moved its backend database from Oracle to Cassandra to gain scalability and reduce costs. The migration strategy involved dual writes to both databases, fork lifting the existing Oracle dataset, and a consistency checker. Challenges included security, denormalization, and engineering effort. Real use cases like APIs and viewing history are discussed along with lessons learned around data modeling, performance testing, and thinking of Cassandra as just storage.
Netflix's Big Leap from Oracle to Cassandra
- 1. NETFLIX’S BIG LEAP FROM ORACLE TO C* ROOPA TANGIRALA Engineering Manager Netflix
- 2. WHO AM I? Engineering Manager @ Netflix Twitter - @roopatangirala Email roopa.tangirala@gmail.com Linkedin - https://www.linkedin.com/pub/roopa-tangirala/3/960/2b
- 3. OVERVIEW • Brief History • Set up • Migration Strategy • Migration Challenges • Example of Real use cases • Lessons learnt
- 4. 1997
- 5. DATACENTER
- 6. BACKEND
- 9. LICENSE COST
- 11. 2008
- 12. MOVE TO CLOUD
- 13. REQUIREMENTS • HIGHLY AVAILABLE • MULTI DATACENTER SUPPORT • PREDICTABLE PERFORMANCE AT SCALE
- 14. WHY C* ? • Massively scalable architecture • Multi-datacenter, multi-directional replication • Linear scale performance • Transparent fault detection and recovery • Flexible, dynamic schema data • Guaranteed data safety • Tunable data consistency
- 15. BACKEND NOW
- 16. MICRO SERVICES • Horizontal, Homogenous, Commoditized
- 17. DOWNTIME
- 19. ACTIVE ACTIVE
- 20. GLOBAL PRESSENCE
- 22. BABY STEPS
- 23. NEW FEATURES FIRST TO CLOUD
- 24. DATA MODEL REVIEW keyspace column family Row column •name •value •timestamp DB/Schema Table Row column • name • value O R A C L E CASSANDRA
- 25. SCHEMALESS DESIGN • Row-oriented • Number of columns/Names can differ namexyz Paul zip 95123 nameabc Adam zip 94538 sex Male namecde XYZ
- 26. UNDERSTAND WRITE PATH client Commit log (Disk) Memtable (memory) sstable sstable sstable Flush
- 27. UNDERSTAND READ PATH client memtable sstable sstable sstable Row cache/key cache
- 28. LOGIC IN APPLICATION • Stored procedures • Functions • Triggers • Referential integrity constraints
- 30. MAIN APPROACH • DUAL WRITES • FORKLIFT OLD DATASET • CONSISTENCY CHECKER • MIGRATE READS TO C* FIRST • SWITCH OFF WRITES TO DC
- 31. Relationships – Better in App Layer
- 32. ORACLE SEQUENCE • USE UUID for Unique keys • Distributed Sequence Generator for Ordering • C* counters – not so much
- 33. Heavy Transactional Use Case RDBMS
- 34. CHALLENGES
- 35. SECURITY
- 37. Roman Riding
- 40. Know limitations
- 41. SOURCE OF TRUTHF TRUTH
- 42. REAL EXAMPLES
- 43. API • High concurrency • Range scans • ~1MB of data • Caused heap issues at Cassandra level • Very high read latency
- 44. Range scan replaced with inverted index 0000/odp/{ui}/pathEvaluator_2 active Scripts_1 Text datafalseallocation 0000 /xbox/{ui}/pathEvaluator_ 1 active Scripts_1 Text datafalseallocation 0000/tvui/{ui}/pathEvaluator_1 active Scripts_1 Text datafalseallocation active_Scripts_idx 1, 2 Idx_1 /tvui/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator 2/odp/{ui}/pathEvaluator scripts client 1 2
- 45. Inverted Index considerations • Column name can be used a row key placeholder • Hotspots!! • Sharding
- 46. VIEWING HISTORY
- 47. Growth of Viewing History 47
- 48. Problem Growth Pattern: “Hockey-stick” Retention Policy: Retain forever Access Pattern: Retrieve all for a customer Scaling and performance challenges as the data grows 48
- 49. Goals Small Storage Footprint Consistent Read/Write Performance Infinite Scalability 49
- 51. Old Data Model - Pros Simple Wide Rows No additional processing Fast Write Fast Read for Majority 51
- 52. Old Data Model - Cons Read latency increases as number of viewing records increases Cassandra Internal • Number of SSTables • Compaction • Read Repair Lesson learned : Wide Row Limitation 52
- 53. New Data Model Split Viewing History into two Column Families 1. Recent • Small records with frequent updates • Cassandra tuning : compaction, read repair, etc. 2. Compressed • Large historical records with rare updates • Rollup • Compression • Cassandra: rare compaction, no read repair 53
- 54. New Data Model cont’d 54
- 55. RESULTS 55
- 56. Think Data Archival Data stores grow exponentially Have a process in place to archive data • Moving to a separate column family • Moving to a separate cluster (non SSD) • Setting right expectations w.r.t latencies with historical data Cassandra TTL’s
- 57. Cinematch Rating Service • First Model • Second Model Movie_id12345 BLOB Movie_id4355 BLOB 545 1 545 4 545 534512345 4 545 2 454355 2 66 2 67 4
- 58. WHAT WORKED? 12345 4355 343 674 5 Name 3 4542443 242 Name BLOB 4 BLOB52
- 59. BLOB vs COLUMN/VALUE Try out column/value approach first and hopefully it satisfies avg/95/99th Column value pro’s: • Write payload can be smaller • Query by specific columns • Read path does not require reading the entire row Blob considerations • Read percentage is very high (90’s) • Read latencies are very important • All the data is read most of the time
- 60. LESSONS LEARNT • Get the model right • Baby Steps • Huge Engineering effort • Think Compression • Performance test • DB just for Data
- 61. WE ARE HIRING !! – CHECK OUT JOBS.NETFLIX.COM
Editor's Notes
- #8: RDBMS not flexible enough to handle modern day applications, which are a mixture of structured, semi-structured and unstructured data. WHILE Oracle handles structured data well it lacks dynamic datamodel which is important for high traffic modern applications.
- #9: Oracle scales up , master slave design limits both its scalability and performance for online applications, the failure of Oracle to add more capacity online in an elastic fashion without impacting the applications is a major drawback. IBM power 7 , 32 dual core 64 bit multiprocessors, 750GB RAM, TB’s storage allocated via SVC MCS cluster which is 8G4 cluster with 4GB/sec interface running with RAID 10 configurations.
- #11: Website went down every two weeks to include changes to PLSQL procedures, packages, schema changes and other backend database changes. Was ok for DVD since the shipping could happen after the site was back up but not acceptable for streaming service.
- #13: 1. Re-architect – question everything including keeping our own DC 2. Letting AWS focus on DC allows our engineers to focus on building and improving our business 3. Ideal for horizontal scaling architecture – even if we are not able to predict the customer growth 4. Cloud computing is the future.
- #25: Rdbms backgroud Keyspace -> DB CF -> Table Row groups columns Each column is a triplet Column naming is not necessary/could be different. Column comparator specifies the sorting.. No need to stick to certain rules Name -> sorted Timestamp -> conflict resolution
- #26: Rows are indexed Columns are sorted based on comparator you specify, so use it to your benefit Keep column names short as they are repeated Column size = 15 bytes + size of name + size of value Don’t store empty columns if there is no need – schema free design
- #27: Commit log for durability – sequential write Memtable – no disk access (no reads or seeks) Sstables written sequentially to the disk
- #28: 2 digest query/ 1 complete data response. The optimization is only on the bandwidth Number of replicas contacted depend on the consistency level specified Hinted handoff, read repair, antientropy node repair
- #44: API servers autoscale or new push, they need to read majority of rows in scripts column family
- #45: Simple but powerful concept – based on premise that rows are indexed and point looks are faster Create another column family and store list of all required rowid’s for faster lookup
- #46: Wide row can reside only on one node.. And that can create hot spots Sharding – application logic / buckets
- #47: Each customer id we have a row in cassandra Keep appending columns as and when customer generates viewing history
- #51: Viewing history for one customer is stored in one wide row with row key CustomerID in cassandra. Write One viewing record is inserted as a new column when a customer starts playing a movie (aka Open a stream). Viewing record is updated after customer pauses or stops related playback. Read Read whole row at once to retrieve all viewing records for one customer. This is efficient initially when number of records per customer is small. Read whole row via pagination for large viewing history read latency increases as number of viewing records increases
- #57: We don’t have linear growth TTL fascinating feature… coming from oracle background