Lighthouse - an open-source library to build data lakes - Kris Peeters
0 likes355 views
Lighthouse is an open-source toolkit designed to build data lakes, enabling effective data management through batch and real-time processing. It facilitates data exploration, pipeline construction, and structured storage, while supporting various data sources and analytics needs. The toolkit is particularly beneficial for organizations looking to enhance their big data projects and streamline data reuse across different use cases.
Lighthouse - an open-source library to build data lakes - Kris Peeters
- 1. Lighthouse an open-source toolkit to build data lakes Kris Peeters
- 3. Wow, you nerds are really on to something.
- 5. How did you clean the input data? Can I see the logs? Can you run that model every night? Can I explore that data real quick? Can you run the model live on our website? Can we also start loading clickstream and IoT data? Can you add this data source from marketing? Can I reuse that data for another crazy idea I’m having? What’s the uptime SLA on that thing?
- 8. Batch processing Real-time processing DATA BROKER INSIGHT INSIGHT Serving layer DATABASE API ALERTS Data exploration & data science DATA LAKE A big data system needs the following capabilities
- 9. Data exploration & data science Batch processing Real-time processing Serving layer Putting some technologies on each capability S3 Athena Python RDS Lambda API gateway
- 10. Batch processing Real-time processing DATA BROKER INSIGHT INSIGHT Serving layer DATABASE API ALERTS Data exploration & data science DATA LAKE A big data system needs the following capabilities
- 11. Store first Ask questions later
- 12. Reuse data across use cases
- 14. Lighthouse is an open-source toolkit to build data lakes
- 16. DAS WAR EIN BEFEHL!
- 17. Your code is really poor quality sir Your test coverage is low sir Configuration in json? What is this? 1923? You didn’t even define clear interfaces sir
- 18. Can I at least help build the new version?
- 19. No No No Nope nope nope nope nope nope nope nope nope
- 20. Lighthouse is an open-source toolkit to build data lakes
- 21. Define Data lake Construct Data pipelines Batteries Included* *pull requests welcome
- 22. 90% of big data projects Welcome to my swamp
- 23. / /clean /master /metrics /models Clickstream CRM Customer Service ERP Invoicing Social media Customer360 Asset utilisation Factory efficiency Product revenue Web sessions Customer Loyalty Stock levels Churn prediction Next product to buy Asset optimisation preventative maintenance
- 24. I have a bunch of raw data sources, in this case CSV I will clean them and store them in the default format, ORC Finally, I will build a single view of airplanes, and I will make it available in Hive
- 25. Uniquely identifying and reusing data sources (from HDFS, Hive, JDBC, …) Structured storage formats such as ORC Consistent reads and writes to the data lake
- 27. Clean each data source Combine the two weather data sources Build a single view of airplanes, including weather data
- 28. Structured way of building data pipelines Each individual transform is easy to test Separate how to get the data from what you do with it
- 29. Parameter store - Store credentials and config parameters - Use AWS Parameter Store or Filesystem Vault Testing functions - Compare datasets - Pretty Print data frames - Compare columns - spark unit tests Demo project - Simple setup - Data lake & data pipelines - Copy&paste for your needs Batteries included
- 35. - If you have more than one project in Spark that you need to implement AND has to run in production - If you have business analysts and data scientists who want to explore and play with all the data that is available in the organisation - If you don’t have a lot of experience yet - If you only have to run one very particular use case. Lighthouse will generate more overhead - If you don’t have a large volume or variety of data. Python + Postgres will do just fine - If you are in prototyping / experimentation mode, and there is no need yet to put stuff in production
- 36. Lighthouse an open-source toolkit to build data lakes Kris Peeters https://github.com/datamindedbe/lighthouse