How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Streams And Actors
11 likes5,175 views
Dustin Lyons, an engineering manager at Credit Karma, discusses the evolution of the company's data infrastructure and the implementation of Akka Streams for real-time decision-making. The document highlights how Akka Streams improved data throughput and performance, addressing challenges like backpressure and JVM memory issues. Key results include reduced service level agreements for data delivery and enhanced overall system efficiency.
How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Streams And Actors
- 1. 1 Proprietary & Confidential1 Proprietary & Confidential Using Akka Streams For Real Time Decision Making Dustin Lyons Engineering Manager, Data Platform
- 2. 2 Proprietary & Confidential ● Engineer turned Engineering Manager at Credit Karma ● Data & Analytics on the Platform team ● Build things that make decisions on where data should go ● Lover of science fiction, sushi, and electronic music Who I am
- 3. 3 Proprietary & Confidential Credit Karma is a free financial assistant, helping over 60 million people make progress.
- 4. 4 Proprietary & Confidential 1. Data Infrastructure at Credit Karma: Past and current 2. Mo’ data, mo’ problems 3. Akka Streams saves the day 4. Results and learnings 5. Q&A Agenda for today
- 5. 5 Proprietary & Confidential Data scale (MB/min) @ Credit Karma
- 6. 6 Proprietary & Confidential Credit Karma data platform: PHP days PHP Scripts
- 7. 7 Proprietary & Confidential New tools to help with scale
- 8. 8 Proprietary & Confidential Credit Karma data platform: Scala in 2014 Data Warehouse Import
- 9. 9 Proprietary & Confidential New tools to help with concurrency
- 10. 10 Proprietary & Confidential Credit Karma data platform: Akka in 2015 Analytics Export Service + Data Warehouse Import
- 11. 11 Proprietary & Confidential Credit Karma data platform: Akka in 2015 Analytics Export Service + Data Warehouse Import
- 12. 12 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 13. 13 Proprietary & Confidential Analytics export service
- 14. 14 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 15. 15 Proprietary & Confidential Analytics export service
- 16. 16 Proprietary & Confidential Data warehouse import ReaderDeduplicatorProcessor Extractors Data Warehouse Import Service
- 17. 17 Proprietary & Confidential Data warehouse import
- 18. 18 Proprietary & Confidential Marble maze
- 19. 19 Proprietary & Confidential Marble maze
- 20. 20 Proprietary & Confidential Marble maze
- 21. 21 Proprietary & Confidential Marble maze
- 22. 22 Proprietary & Confidential Marble maze 1Reading from file
- 23. 23 Proprietary & Confidential Marble maze 1 2 Reading from file Waiting for external service
- 24. 24 Proprietary & Confidential Marble maze 1 3 2 Reading from file Objects sit in heap Waiting for external service
- 25. 25 Proprietary & Confidential Marble maze 1 3 2 Reading from file Objects sit in heap Waiting for external service 4 Database Insert
- 26. 26 Proprietary & Confidential Backpressure
- 27. 27 Proprietary & Confidential What is backpressure? Backpressure refers to the buildup of data at an I/O switch when buffers are full and not able to receive additional data. No additional data packets are transferred until the bottleneck of data has been eliminated or the buffer has been emptied.
- 28. 28 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 29. 29 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 30. 30 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 31. 31 Proprietary & Confidential Data warehouse import ReaderDeduplicatorProcessor Extractors Data Warehouse Import Service
- 32. 32 Proprietary & Confidential Akka Streams: Backpressure in action Actor Actor Data Demand
- 33. 33 Proprietary & Confidential Akka Streams: Creating a stream Source Flow Sink
- 34. 34 Proprietary & Confidential Akka Streams: Built in stages Built In Sources • actorRef • actorPublisher • fromIterator • fromFile • Apply (from a Seq) Built In Processing Stages • map • filter • grouped • drop/take • dropWhile/takeWhile • sliding Built In Sinks • head • last • seq • foreach • actorRef • actorSubscriber • reduce • fold Backpressure Aware Stages • mapAsync • buffer (Backpressure) • batch • buffer (Drop) • buffer (Fail) Reference: http://doc.akka.io/docs/akka/current/scala/stream/stages-overview.html
- 35. 35 Proprietary & Confidential Analytics export service Coordinator Data Transformer Workers Kafka Importer Workers Analytics Export Service HTTP Ingest Server
- 36. 36 Proprietary & Confidential Analytics export service Coordinator Analytics Export Service HTTP Ingest ServerAkka Stream
- 37. 37 Proprietary & Confidential Analytics export service
- 38. 38 Proprietary & Confidential Data warehouse import ReaderDeduplicatorProcessor Extractors Data Warehouse Import Service
- 39. 39 Proprietary & Confidential Data warehouse import Extractors Data Warehouse Import Service Akka Stream
- 40. 40 Proprietary & Confidential Data warehouse import service
- 41. 41 Proprietary & Confidential Analytics export service heap (before) GiB=> Time => 28 GiB Red: Heap Space Blue: Used Heap Space Purple: Max Heap Space
- 42. 42 Proprietary & Confidential Analytics export service heap (after) GiB=> Time => 28 GiB
- 43. 43 Proprietary & Confidential Data warehouse import
- 44. 44 Proprietary & Confidential Data warehouse import
- 45. 45 Proprietary & Confidential Data warehouse import
- 46. 46 Proprietary & Confidential • Akka Streams allowed us to move data with increased throughput and optimal performance • No longer getting paged for JVM out of memory or spending time tuning our services • Reduced the SLA for data delivery to our business stakeholders Final results
- 47. 47 Proprietary & Confidential • Akka Actors: Great for low latency • Akka Streams: Optimized for high throughput and solving back pressure • Built on top of Akka Actors • Don’t try to build high throughput systems with an actor system, you’ll just start building Akka Streams Lessons learned
- 48. 48 Proprietary & Confidential48 Proprietary & Confidential Thank you! Q&A Dustin Lyons Engineering Manager, Data Platform