Silicon Valley Data Science: From Oracle to Cassandra with Spark
3 likes691 views
The document describes Allant Group's process of migrating their customer recognition application from an Oracle database with JMS messaging to Cassandra with Spark. They introduced Cassandra to replace the Oracle database, but needed to re-architect the application layer to improve scalability and throughput. Allant employed Hadoop and then Spark with the Datastax Cassandra connector to distribute the application logic. Benchmark testing showed throughput increased from 2.5 million to over 125 million records per hour, reducing processing time from 6-7 days to just 3 hours. The migration enabled elastic scaling, higher data throughput, and lower costs while maintaining the existing application logic.
![@TheAllantGroup | @SVDataScience 32
© 2015. ALL RIGHTS RESERVED.
PERFORMANCE BENCHMARK RESULTS - SPARK
Environment Results
JMS – Oracle 2.5 Million / Hour
MapReduce – Cassandra 45 Million / Hour
Spark – Cassandra 125 Million / Hour
[185 Million / Hour for “match only”]
Benchmark 1: Larger Input (~400 Million Profiles)
~50x
From 6-7 days down to ~3 hours!](https://image.slidesharecdn.com/fromoracletocassandrawithsparkcassandrasummit2015svds1-151002225852-lva1-app6891/85/Silicon-Valley-Data-Science-From-Oracle-to-Cassandra-with-Spark-32-320.jpg)
Silicon Valley Data Science: From Oracle to Cassandra with Spark
- 1. FROM ORACLE TO CASSANDRA WITH SPARK
- 2. @TheAllantGroup | @SVDataScience 2 © 2015. ALL RIGHTS RESERVED. WHO ARE WE? Shambho Krishnasamy Fausto Inestroza
- 3. @TheAllantGroup | @SVDataScience 3 © 2015. ALL RIGHTS RESERVED. CUSTOMER RECOGNITION Challenges in the Digital age – Scalability – Throughput – Cost
- 4. @TheAllantGroup | @SVDataScience 4 © 2015. ALL RIGHTS RESERVED. CUSTOMER RECOGNITION Key Management Tailoring Key Assignment Hygiene Func6onal Buckets
- 5. @TheAllantGroup | @SVDataScience 5 © 2015. ALL RIGHTS RESERVED. CUSTOMER RECOGNITION Key Management Tailoring Key Assignment Hygiene Func6onal Buckets
- 6. @TheAllantGroup | @SVDataScience 6 © 2015. ALL RIGHTS RESERVED. LEGACY APPLICATION Input Output Recogni6on Bus Service Party Address HHLD Indv Digital Keying Lookup Reference Address Household Individual DigitalKey Digi-‐Asso Reference
- 7. @TheAllantGroup | @SVDataScience 7 © 2015. ALL RIGHTS RESERVED. LEGACY SOLUTION JMS
- 8. @TheAllantGroup | @SVDataScience 8 © 2015. ALL RIGHTS RESERVED. NEED FOR CHANGE? RE-‐PLATFORM ! RE-‐ARCHITECT !
- 9. @TheAllantGroup | @SVDataScience 9 © 2015. ALL RIGHTS RESERVED. LIMITATIONS TO SCALE – MESSAGE PROCESSING ARCHITECTURE • Message processing engine • Common API to handle real-time and batch • Batch is converted into messages
- 10. @TheAllantGroup | @SVDataScience 10 © 2015. ALL RIGHTS RESERVED. LIMITATIONS TO SCALE – DATA THROUGHPUT 4-‐8 MM records/hour Volume Performance Scale to meet Allant’s Audience Interconnect® customer recogni6on needs
- 11. @TheAllantGroup | @SVDataScience 11 © 2015. ALL RIGHTS RESERVED. LIMITATIONS TO SCALE – SCALING HORIZONTALLY Locking!
- 12. @TheAllantGroup | @SVDataScience 12 © 2015. ALL RIGHTS RESERVED. LIMITATIONS TO SCALE – SCALING VERTICALLY =
- 13. @TheAllantGroup | @SVDataScience 13 © 2015. ALL RIGHTS RESERVED. WHAT DO WE WANT? Increase throughput Improve scalability Elas6c infrastructure (but don’t compromise on real-‐6me API capability!) (but contain cost!) (well… so we went Cloud)
- 14. @TheAllantGroup | @SVDataScience 14 © 2015. ALL RIGHTS RESERVED. WHAT TO RE-PLATFORM? ? JMS
- 15. @TheAllantGroup | @SVDataScience 15 © 2015. ALL RIGHTS RESERVED. CASSANDRA
- 16. @TheAllantGroup | @SVDataScience 16 © 2015. ALL RIGHTS RESERVED. Consistent Reads! Consistent Writes! SWITCH DATA STORE JMS
- 17. @TheAllantGroup | @SVDataScience 17 © 2015. ALL RIGHTS RESERVED. WE’RE DONE!
- 18. @TheAllantGroup | @SVDataScience 18 © 2015. ALL RIGHTS RESERVED. BUT…APPLICATION LAYER IS STILL A BOTTLENECK
- 19. @TheAllantGroup | @SVDataScience 19 © 2015. ALL RIGHTS RESERVED. MUST MAINTAIN EXISTING LOGIC!
- 20. @TheAllantGroup | @SVDataScience 20 © 2015. ALL RIGHTS RESERVED. RECAP…
- 21. @TheAllantGroup | @SVDataScience 21 © 2015. ALL RIGHTS RESERVED. RECAP – BASELINE JMS
- 22. @TheAllantGroup | @SVDataScience 22 © 2015. ALL RIGHTS RESERVED. RECAP -‐ LEGACY APPLICATION Input Output Recogni6on Bus Service Party Address HHLD Indv Digital Keying Lookup Reference Address Household Individual DigitalKey Digi-‐Asso Reference
- 23. @TheAllantGroup | @SVDataScience 23 © 2015. ALL RIGHTS RESERVED. HOW DID WE DO IT?
- 24. @TheAllantGroup | @SVDataScience 24 © 2015. ALL RIGHTS RESERVED. JMS INTRODUCED CASSANDRA… But Cassandra is very bored…
- 25. @TheAllantGroup | @SVDataScience 25 © 2015. ALL RIGHTS RESERVED. What about this part? HOW TO RE-ARCHITECT? JMS Cassandra is very bored…
- 26. @TheAllantGroup | @SVDataScience 26 © 2015. ALL RIGHTS RESERVED. NOW INTRODUCE HADOOP We employed Distributed Data Management Technology end-‐to-‐end… Cassandra is very happy!
- 27. @TheAllantGroup | @SVDataScience 27 © 2015. ALL RIGHTS RESERVED. PERFORMANCE BENCHMARK RESULTS - ENVIRONMENT • 12 Cassandra Nodes – 4 CPU – 15GB RAM – 80GB SSD • 6 Hadoop Nodes – 32 CPU – 60GB RAM – 640GB SSD
- 28. @TheAllantGroup | @SVDataScience 28 © 2015. ALL RIGHTS RESERVED. PERFORMANCE BENCHMARK RESULTS - MAPREDUCE Environment Results JMS – Oracle 4.5 Million / Hour MapReduce – Cassandra 44 Million / Hour Benchmark 1: Smaller Input (~15 Million Profiles) ~10x Environment Results JMS – Oracle 2.5 Million / Hour MapReduce – Cassandra 45 Million / Hour Benchmark 1: Larger Input (~400 Million Profiles) ~20x From 6-7 days down to ~8 hours!
- 29. @TheAllantGroup | @SVDataScience 29 © 2015. ALL RIGHTS RESERVED. COULD WE DO BETTER?
- 30. @TheAllantGroup | @SVDataScience 30 © 2015. ALL RIGHTS RESERVED. INTRODUCE SPARK Cassandra is ecsta6c!
- 31. @TheAllantGroup | @SVDataScience 31 © 2015. ALL RIGHTS RESERVED. EMPLOY DATASTAX LIGHTNING FAST CONNECTOR
- 32. @TheAllantGroup | @SVDataScience 32 © 2015. ALL RIGHTS RESERVED. PERFORMANCE BENCHMARK RESULTS - SPARK Environment Results JMS – Oracle 2.5 Million / Hour MapReduce – Cassandra 45 Million / Hour Spark – Cassandra 125 Million / Hour [185 Million / Hour for “match only”] Benchmark 1: Larger Input (~400 Million Profiles) ~50x From 6-7 days down to ~3 hours!
- 33. @TheAllantGroup | @SVDataScience 33 © 2015. ALL RIGHTS RESERVED. TAKEAWAYS • We did contain cost! – with be^er throughput & scalability • Pu`ng Cassandra to work by employing MapReduce and Spark • Unimpeded throughput regardless of the data-‐store volume • Unique Key Genera6on under distributed data technology • Resolving Latency vs. Throughput -‐ Tradi6onal Conflict • In our use-‐case, the data-‐store • Is encapsulated ! • Has only controlled access ! • Does only Reads and Writes !
- 34. @TheAllantGroup | @SVDataScience 34 © 2015. ALL RIGHTS RESERVED. THANK YOU