Dairy data warehouse - Introducing the concept of Data Science and Big Data in a traditional data warehouse company
- 1. INTRODUCING THE CONCEPTS OF DATA SCIENCE AND BIG DATA IN A TRADITIONAL DATA WAREHOUSE COMPANY Joris Bos Manager IT, DDW
- 2. Dairy Data Warehouse (DDW) • Start up founded in 2013 • Based in Assen, Netherlands • DDW offers data services to dairy farm consultants and corporate stakeholders in the dairy industry across the globe
- 3. DDW BIG Dairy Data Platform 1. DATA INPUT Herd management Data capture devices Milk recording data 2. DATA TRANSFORMATION 3. DEEP LEARNING DDW ETL unique technology transforms all data into a standard common format Dairy Data Warehouse Single consistent database Production Reproduction Disease 4. DDW PREDICTIONS
- 4. ‘Advance to go’ • Daily uploads (batching) • Slow database XML exports • Calculations freezing data scientist’ pc • Valuable data locked away in private networks, widely spread • This new ‘thing’!
- 5. Data warehouses are dead! Or? • Trendy, all will be superseded • New paradigm • On premises or Cloud? • Clearly differentiate use-cases, strengths • HDFS != DMS • Different target audience • Aim to retain value from previous investments
- 6. DDW Big data infrastructure • Set-up from zero, best practices opportunity • Security, data access and data governance at the core • New roles and competences within the organization • Data lake at the center • Seperation storage / compute • Support for persisted, historical data now (batch) • Streaming (real-time) near future • Audit, audit, audit, audit
- 7. Data Lake & Data Warehouse
- 8. Big Data pipeline Data Engineer Data Scientist Value
- 9. Chosen setup (Azure) • Storage : Azure Data Lake (Active Directory Security / Office365) • Integration runtimes (Data Factory) • ELT • Data warehouse as ‘just’ another data source • Data Catalog • Compute : On-demand clusters with Hortonworks distribution platform (HDP) • ML Deep Learning : Model calculation with GPU powered VMs • ETL to data warehouse for hot layer going forward
- 10. Chosen setup
- 11. Takeaways • Establish clear language around Big Data and tooling • Clearly position Big Data in your company (vision, budget, etc.) • Define new roles, workflows and conventions • Stick to best practices • Start small, Determine Minimum Viable Product • Cloud is great for ad-hoc, scalability demands • Iterate and experiment through POCs
- 12. Thank you Feel free to contact me: Joris Bos J.bos@dairydatawarehouse.com