MinneBar 2013 - Scaling with Cassandra
- 1. Jeff Bollinger – CTO - @jbollinger Jeff Smoley – Infrastructure Architect Scaling With Cassandra
- 2. About NativeX The Backstory Why Cassandra Cassandra Overview NativeX Cassandra Implementation / Metrics What we Learned Agenda
- 3. Formerly W3i Marketing technology platform that enables developers to build successful businesses around their apps. NativeX
- 4. Over 620M unique devices on our network Over 500 apps in network > 100M Monthly Active Users 100 GB of data ingest per week Vanity Metrics
- 5. A growing mobile advertising network Backstory 0 1 2 3 4 5 6 2011 Q4 2012 Q1 2012 Q2 2012 Q3 2012 Q4 2013 Q1 Billions API Requests
- 6. Infrastructure Intensive Model 0 2 4 6 8 10 12 0 1 2 3 4 5 6 7 8 9 10 11 12 Millions Session Calls by Week After User Acquired Lifetime of user
- 7. Microsoft SQL Server 2 Node Cluster (failover) 12 cores / node 192 GB of / node Compellent SAN 172 Disk (SSD,FC,SATA) Scale Up Architecture
- 8. Consistency Partition Tolerance Availability CAP Theorem SQL Server, MySQL Cassandra MongoDB
- 9. Scale •Horizontal •Incremental cost structure Resiliency •No single point of failure •Geographically distributed Objectives
- 10. Web Application Tier Database Tier What Needed to Scale Web Application Tier is already a server farm that can scale horizontally through our VMWare environment. Database Tier was one giant monolithic Microsoft SQL Server machine.
- 11. Stands for Not Only SQL The NoSQL movement is not about silver bullets and black boxes. It’s about understanding problems and focusing on solutions. It’s about using the right tool for the right problem. What is NoSQL?
- 12. Selecting Cassandra DB Distributed Maturity High Availability Style Documentation Native Language Drivers Popularity MongoDB Yes Medium Yes Document - NoSQL Excellent Major Languages High VoltDB Yes Low Yes RDBMS - SQL Good Major Languages Low MySQL Cluster Yes High Yes RDBMS - SQL & Key/Value Excellent Major Languages Medium MySQL ScaleDB Yes Low Yes RDBMS - SQL Good Major Languages Low Cassandra Yes Medium Yes Key/Value - Column Family Excellent Major; Poor .Net High CouchDB No Medium Yes Document - NoSQL ? No - REST only Medium RavenDB Yes? Low No Document - NoSQL Poor C#, JS, REST Medium Couchbase Yes Medium Yes Key/Value - Document Good Major Languages Medium *Disclaimer, this data was complied in spring of 2012 and my not reflect the current state of each database system shown here. http://nosql.mypopescu.com/ is a helpful site for discovering and learning about different DB Systems.
- 13. Considered Multiple DB Providers MySQL Cluster Relational and very familiar. Has physical row limitations. MongoDB Data modeling was simpler than C*. Not very clear if it had multi-cluster support. Cassandra At the very core it’s all about scalability and resiliency. Data modeling a little scary, limited .Net support. Top Choices
- 15. C* was not a replacement DB system, but an addition. C* solves a very specific problem (for us). Writing large volumes of data quickly. Reading very specific data out of a large record set. NoSQL solutions, like C*, are not meant to be a replacement for everything. You will make your lifer harder if you try! The same should be said about Relational Databases. They don’t solve every problem! C* at NativeX
- 16. We have three major classifications of data. Configuration Activity Tracking Device History Data Classification
- 17. This data is relatively small in total size and is used to operationally run our products. Examples include: Mobile Apps Offers Campaigns Restrictions Queue Settings This data is typically relational and therefore continues to be stored in MS SQL Server. Configuration Data
- 18. Data is stored inside of Column Families using nested Key/Value pairs. A Row Key maps to a collection of Columns. A Column Name (AKA Column Key) maps to a Column Value. The Column Name is stored along side the Value. A common strategy is to store JSON/XML in the Column Value. (Side note, if you’ve heard of Super Columns, forget about them, they hurt more than they help) The Very Basics of C* Data Modeling
- 19. Raw tracking data for all activities used by the ETL process to produce OLAP data on an hourly basis. Synonymous with Time Series, Event Series, or Logging data. Examples include: Running of Mobile Apps Viewing Offers Clicking on Offers Receiving Rewards Activity Tracking Data
- 20. Historical activities that each device has performed while being part of NativeX’s network. Used for offer classification for a given device. Examples include: Clicking on Offers Running Mobile Apps Redeeming Rewards Device History Data
- 21. 12 Nodes Cisco UCS Blades 12 Cores @ 2.0GHz with Hyper-threading 64GB of Ram 2 x 480GB Intel commodity SSDs in RAID 0 10.5 TB total, ~7 TB usable Red Hat Linux Hardware
- 22. We chose to use Enterprise hardware for the servers so that we would have support for them. However, our work load is very read heavy and 15K rpm rotational disks were a bottle neck. We chose to swap out the rotational for commodity SSDs. (Enterprise SSDs were 10x as expensive) We have limited support on the hardware because of this. Commodity Vs. Enterprise
- 23. 240 peak Writes per second per node 2,880/sec cluster wide 888 peak Reads per second per node 10,656/sec cluster wide 0.53 ms average Write Latency per request 1.7 ms average Read Latency per request Almost 3 TB of data adding 1 TB a month Internal C* Cluster Stats
- 24. MS SQL Writes 12 ms Reads 1.5 ms C* Writes 3 ms Reads 4 ms Application Side Latencies
- 25. We think that in SQL Server, reads were faster because most of the data sat in memory. We might be able to achieve lower latencies in C* if we gave each node just as much memory as our SQL Server. To counter act the increased latencies we used certain techniques like parallel reads using multi- threading in our web application. Can We Make Reads Faster?
- 26. There are still challenges with C*, like any complex system. More moving parts and things that need to stay in sync. Misconfigurations can literally destroy your data. Certain config settings cannot be changed after you are live, such as the number of virtual Racks. Not all Roses
- 27. Get into production early Data Import = Reality Break down communication barriers Understanding your IO profile is really important Cassandra changes quickly, you need to keep up Scalable systems like C* have a massive amount of knobs, you need to know them Leverage cloud resources in working toward right sizing your cluster Lessons Learned
- 28. We’re hiring http://nativex.com/careers/ Join the MSP C* Meetup http://www.meetup.com/Minneapolis-St-Paul-Cassandra- Meetup/ Email us Jeff.Smoley@nativex.com Jeff.Bollinger@nativex.com or @jbollinger Slide Deck http://www.slideshare.net/JBollinger/minnebar-2013-scaling- with-cassandra Thanks