Data vault modeling et retour d'expérience
1 like645 views
The document outlines the implementation of an agile business intelligence (B.I.) project at Evam, detailing its pre-project phase, challenges, and methodologies adopted. Key components include the use of data vault modeling for data warehousing, agile project management principles, and a focus on continuous improvement and data quality. The document also emphasizes the importance of collaboration, stakeholder involvement, and effective project management practices throughout the project timeline.
Data vault modeling et retour d'expérience
- 1. BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH Agile Business Intelligence @ Evam
- 2. Plan • Introduction ( F. Kang à Birang) • Pre-project (F. Kang à Birang & J-M. Delacrétaz) • Agile project management (A. Martino) • Agile architecture (E. Fidel) • Data quality (A. Martino) • EVAM Feedback (B. Albietz)
- 3. Introduction Fabienne Kang à Birang – Business Analyst / Product owner
- 4. Introduction • EVAM Presentation • Project Sponsor • Director • Indicators • 2013 – Existing B.I.
- 5. Pre-Project Phase Fabienne Kang à Birang – Business Analyst / Product owner Jean-Marc Delacrétaz – Developer
- 6. Pre-Project Phase • Target • Operational reporting • Problems encountered @ EVAM • Data interpretation • Business rules errors • Prerequisites • Dictionary • Population hierarchized
- 7. Preexisting B.I. • 2013 • P.O.C. to introduce B.I. «philosophy» • Chosen Tools • ETL : Talend • Reporting : Tibco JasperReport • Weaknesses • Lack of expertise & methodology • Bad performances
- 8. Decision in August 2014 • Start from scratch • With Trivadis Lausanne as a partner • Tools • Performances • Architecture with « Best practices »
- 9. Agile Project Management Adriano Martino – Senior B.I. Consultant
- 10. Agility We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan
- 11. Organisation • Evam• Evam • Trivadis • Evam• Trivadis Scrum Master Product Owner CustomerDeveloppers
- 12. Agile Objectives • Deliver working software frequently • Adapt to change
- 13. Scrum components overview Sprint Planning Sprint Backlog Product Backlog Daily Stand up Sprint 2 to 4 weeks Sprint Review Retrospective
- 14. Normal Process for a B.I. need Business Analysis Design of the model Implementation Unit Testing Volume testing User Acceptance Testing New Need Rework Rework Rework Rework Deployment to Validation Deployment Production
- 15. Normal Process for a B.I. need
- 16. Agile Objectives • Adapt to change • Deliver working software frequently • At regular intervals, the team reflects on how to become more effective
- 17. Cadence SCRUM EVENT DRIVEN Sprint1 Sprint2 Sprint3 … RetrospectiveReviewReleasePlanning1 2 1 3 4 2 3 4 1 2 3 4 1 2 3 4 1 2 2 2 2 213 42
- 18. Agile Objectives • Adapt to change • Deliver working software frequently • At regular intervals, the team reflects on how to become more effective • Work close to business
- 19. Collaborative Workshops Business Need analysis Technical analysis Live dev Prototyping Live testing
- 20. Agile B.I. Architecture • Evolutive • Easy change management • Parallelisable development • Business oriented • Integration • Possibility to automate generation We choose Data Vault Modelling
- 21. Agile Architecture Eddie Fidel – Senior B.I. Consultant
- 22. STAGING DYNAMIC ETL Enterprise Data Warehouse With data vault Modeling Agile Bi Architecture SOURCES Virtualized Data Marts
- 23. STAGING DYNAMIC ETL Enterprise Data Warehouse With data vault Modeling Data Warehouse Layer SOURCES Virtualized Data Marts DYNAMIC ETL
- 24. What is Data Vault ? • Data Modelling Method for Data Warehouses in Agile Environments • Developed by Dan Linsted • Suitable for • DWH Core Layer • Optimized for • Agility / Integration / Historization
- 25. Data Vault composition • Decomposition of Source Data • Split Data into Separate Parts Hubs Business Entity Links Relations Satellites Contexts Business Oriented
- 26. Data Vault composition • Elements : Hub – Link – Sat Customer Sat Sat Sat Customer Product Sat Sat Sat Product Hub = List of Unique Business Keys Link = List of Relationships, Associations Satellites = Descriptive Data Order Sat Sat Sat Order Link
- 27. Avantages and challenges • Standard ETL Rules to Load Data Vault • Easy Extensibility of Data Vault Model • Integration of Multiple Source Systems • Traceability and Complete History • High Number of Tables in Data Vault
- 28. What does the Data Vault generator do ? • Tables • Indexes • Surrogate keys • Foreign keys • Partitions • Loading process • SCD1 / SCD2 • Loading audits • Handling Errors
- 29. Generator value 29 Business spec Technical spec Development Test Deployment Qualityassurance Documentation Simplify Generator Documentation QS Total Savings Fast and short implementation cycles Broad flexibility of change Auto-generated quality assured components Huge time and cost savings On-going and recurrent with each step of modification or enlargement!!!
- 30. STAGING DYNAMIC ETL Enterprise Data Warehouse With data vault Modeling Dynamic ETL SOURCES Virtualized Data Marts
- 31. Dynamic ETL for DWH • Parallel Loading • HUB • LINK et SAT • Dynamic call to loading procedures • No deployment of ETL needed
- 32. STAGING DYNAMIC ETL Enterprise Data Warehouse With data vault Modeling Dynamic ETL SOURCES Virtualized Data Marts DYNAMIC ETL
- 33. Data Mart • Business Need Oriented • Virtualized DM (materialized view) • Can be regenerated from scratch • Find value at a point in time • Good perfomance • Automatically regenerated (no deployment)
- 34. Data Quality Adriano Martino – B.I. Consultant
- 35. Quality report • Automated • Daily execution • Simple development • Possible to send mail based on result • Direction support to involve Business
- 36. EVAM Feedback Bruno Albietz – I.T. Manager
- 37. Keys Learnings • Show business value as early as possible and keep the ball rolling • Project: December 2014 – June 2016 • Phased implementation: 1st output in June 2015, then regular outputs on a monthly basis • Be prepared to spend most of your time on data quality • The lifeblood of B.I. projects
- 38. Keys Learnings • Prepare knowledge transfer to your staff during the project • Modelling, ETL, Reporting • Good project management practice, from business requirements to report development • Increase user buy-in with Scrum • Key users and management involved from day 1
- 39. Keys Learnings • Learn to say “ No ” • B.I. quality versus business process quality • B.I. is also here to show process deficiencies, do not try to solve all business issues within the B.I. project
- 40. Q & A