Migrating Data Pipeline from MongoDB to Cassandra
6 likes2,263 views
Demi Ben-Ari, a senior software engineer, discusses the transition from MongoDB to Apache Cassandra due to performance issues with batch job processing times. The solution involved writing to both databases simultaneously and leveraging Apache Cassandra's scalability and community support, resulting in performance improvements. Key lessons include the importance of monitoring, data modeling, and using prepared CQL statements for efficiency.
Migrating Data Pipeline from MongoDB to Cassandra
- 1. Demi Ben-Ari Cassandra Meetup â 10/11/2015 Israel
- 2. About me Demi Ben-Ari Senior Software Engineer at Windward Ltd. BSâc Computer Science â Academic College Tel-Aviv Yaffo In the Past: Software Team Leader & Senior Java Software Engineer, Missile defense and Alert System - âOfekâ unit - IAF
- 3. Agenda ï Data flow with Mongo DB ï The Problem ï Solution ï Lessons learned from a Newbi ï Conclusion
- 5. Data pipeline flow â Use Case ï Batch Apache Spark applications running every 10 - 60 minutes ï Request Rate: ⊠Bursts of ~9 million requests per batch job ⊠Beginning â Reads ⊠End - Writes
- 6. Workflow with MongoDB Worker 1 Worker 2 âŠ. âŠ. ⊠⊠Worker N MongoDB Replica Set Spark Cluster Master Write Read
- 7. Spark Slave - Server Specs ï Instance Type: r3.xlarge ï CPUâs: 4 ï RAM: 30.5GB ï Storage: ephemeral ï Amount: 10+
- 8. MongoDB - Server Specs ï MongoDB version: 2.6.1 ï Instance Type: m3.xlarge (AWS) ï CPUâs: 4 ï RAM: 15GB ï Storage: EBS ï DB Size: ~500GB ï Collection Indexes: 5 (4 compound)
- 9. The Problem ï Batch jobs ⊠Should run for 5-10 minutes in total ⊠Actual - runs for ~40 minutes ï Why? ⊠~20 minutes to write with the Java mongo driver â Async (Unacknowledged) ⊠~20 minutes to sync the journal ⊠Total: ~ 40 Minutes of the DB being unavailable ⊠No batch process response and no UI serving
- 10. Alternative Solutions ï Shareded MongoDB (With replica sets) ⊠Pros: ï Increases Throughput by the amount of shards ï Increases the availability of the DB ⊠Cons: ï Very hard to manage DevOps wise (for a small team of developers) ï High cost of servers â because each shared need 3 replicas
- 11. Workflow with MongoDB Worker 1 Worker 2 âŠ. âŠ. ⊠⊠Worker N Spark Cluster Master Write Read Master
- 12. Our DevOps â After that solution We had no DevOps guy at that time at all ï
- 13. Alternative Solutions ï DynamoDB (Weâre hosted on Amazon) ⊠Pros: ï No need to manage DevOps ⊠Cons: ï Catholic Wedding Amazons Service ï Not enough usage use cases ï Might get to a high cost for the service
- 14. Alternative Solutions ï Apache Cassandra ⊠Pros: ï Very large developer community ï Linearly scalable Database ï No single master architecture ï Proven working with distributed engines like Apache Spark ⊠Cons: ï We had no experience at all with the Database ï No Geo Spatial Index â Needed to implement by ourselves
- 15. The Solution ï Migration to Apache Cassandra (Steps) ⊠Writing to Mongo and Cassandra simultaneously ⊠Create easily a Cassandra cluster using DataStax Community AMI on AWS ⊠First easy step â Using the spark-cassandra- connector ï (Easy bootstrap move to Spark ïł Cassandra) ⊠Creating a monitoring dashboard to Cassandra
- 16. Workflow with Cassandra Worker 1 Worker 2 âŠ. âŠ. ⊠⊠Worker N Cassandra Cluster Spark Cluster Write Read
- 17. Result ï Performance improvement ⊠Batch write parts of the job run in 3 minutes instead of ~ 40 minutes in MongoDB ï Took 2 weeks to go from âZero to Heroâ, and to ramp up a running solution that work without glitches
- 18. Lessons learned from a Newbi ï Use TokenAwarePolicy when connecting to the cluster â Spreads the load on the coordinators Cluster cluster = null; Builder builder = Cluster.builder() .withSocketOptions(socketOptions); builder = builder.withLoadBalancingPolicy(new TokenAwarePolicy( new DCAwareRoundRobinPolicy())); cluster = builder.build();
- 19. Lessons learned from a Newbi ï Monitor everything!!! â All of the Metrics ⊠Cassandra ⊠JVM ⊠OS ï Feature flag every parameter to the connection, youâll need it for tuning later
- 20. Monitor Everything!!! ï DataStax â OpsCenter ⊠Comes bundled with the DataStax Community AMI on AWS
- 21. Monitor Everything!!! ï Graphite + Grafana ⊠Pluggable metrics â Since Cassandra 2.0.x ï Cassandra internal metrics ï JVM metrics ⊠OS â Metrics ï CollectD / StatsD â Reporting to graphite ⊠Should be combined with application level metrics in the same graphs ï Better visibility on correlations of the metrics
- 22. Monitor Everything!!! ï Graphite + Grafana
- 23. Lessons learned from a Newbi ï ânodetoolâ is your friend ⊠tpstats, cfhistograms, cfstats⊠ï Data Modeling ⊠Time series data ⊠Evenly distributed partitions ⊠Everything becomes more rigid ï Know your queries before you model
- 24. Lessons learned from a Newbi ï CQL Queries ⊠Once we got to know our data model better, It got more efficient performance wise to use CQL statement instead of the âspark-cassandra- connectorâ ⊠Prepared Statements, Delete queries (of full partitions), Range queriesâŠ
- 25. Useful Cassandra GUI Clients ï DevCenter â By DataStax - Free ï Dbeaver â Free & Open Source ⊠Supports a wide variety of databeses
- 26. Conclusion ï Cassandra is a great linear scaling Distributed Database ï Monitor as much as you can ⊠Get visibility of whatâs going on in the Cluster ï Data Modeling correctly is the Key for success ï Be ready for your next war ⊠Cassandra performance tuning â Youâll get to that for sure
- 27. Questions?
- 28. Thanks, Resources and Contact ï Demi Ben-Ari ⊠LinkedIn ⊠Twitter: @demibenari ⊠Blog: http://progexc.blogspot.com/ ⊠Email: demi.benari@gmail.com ⊠âBig Thingsâ Community ï Meetup, YouTube, Facebook, Twitter