Using Spark at Vungle
1 like2,703 views
The document discusses the evolution of an ETL pipeline from an old architecture to a new streaming-based one. The old architecture ran hourly jobs that processed 12+ GB of data and could take over an hour to complete. The new architecture uses streaming to provide horizontal scalability and real-time processing. It decouples ingestion of raw data from processing via Spark streaming. Events are ingested into MongoDB as they arrive and then processed to calculate metrics and output to various destinations.
Using Spark at Vungle
- 1. Introduction Old Architecture New Architecture Decoupling Streaming Conclusion 1
- 2. 2 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion ● Introduction ● Old Architecture ● New Architecture ● Decoupling ● Streaming ● Conclusion
- 3. 3 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion ● Legacy Java Process ○ “Crunches” data ○ Sends data downstream to our own datastores and to 3rd party analytics ○ Runs every hour ● Growth ○ Process can run over an hour ○ 12 GB -> 24GB heap in less than 1 year ○ Cron is a horrible job management system ○ A failure requires rerunning a job from the beginning ● 2.0 ○ Horizontably scalable ○ Real Time ETL ○ Reuesable
- 4. 4 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion ETL @ Vungle ● ~1 Billion Events / Day ● Deduplication ● Calculating $$$ ● Outputting data to various destinations
- 5. 5 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Old Architecture
- 6. 6 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 7. 7 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 8. 8 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 9. 9 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 10. 10 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 11. 11 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 12. 12 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 13. 13 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 14. 14 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion New Architecture
- 15. 15 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 16. 16 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 17. 17 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 18. 18 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 19. 19 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 20. 20 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 21. 21 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 22. 22 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Decoupling
- 23. 23 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 24. 24 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 25. 25 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 26. 26 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 27. 27 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 28. 28 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 29. 29 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 30. 30 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 31. 31 Introduction Problem Decoupling Streaming Conclusion Setup connection and spark streams Map each line of log into Mongo Objects and insert into mongo
- 32. 32 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Setup connection and spark streams
- 33. 33 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Mapping to Mongo objects and insertions
- 34. 34 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Questions
- 35. 35 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Streaming
- 36. 36 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 37. 37 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 38. 38 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 39. 39 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Ingestion
- 40. 40 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Event ID Request View Install ... Request Added View Added Install Added Value Ingestion Table Schema
- 41. 41 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion ... Date Time Deliveries Views Installs Processed Deliveries Processed Views Processed Installs Fact Table Schema
- 42. 42 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Ingestion
- 43. 43 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 44. 44 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 45. 45 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 46. 46 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 47. 47 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 48. 48 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 49. 49 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Process
- 50. 50 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 51. 51 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 52. 52 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 53. 53 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 54. 54 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion
- 55. 55 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Next Steps ● Switching from JSON to ProtoBuf ● Using YARN to run multiple jobs on one cluster ● Data Science ● Who knows?
- 56. 56 Introduction Old Architecture New Architecture Decoupling Streaming Conclusion Questions
- 57. Thank you! 57