Agile Data Architecture
Download as PPT, PDF3 likes2,327 views
The document provides an overview of agile data architecture, highlighting the benefits of adopting agile methodologies for enterprise-level databases and data warehouses. It discusses key principles, practices for evolutionary design, and the importance of collaboration between developers and DBAs. The content emphasizes the need for continuous integration and refactoring to improve data architecture and overall project success.
Agile Data Architecture
- 1. Agile Data Architecture Overview • Tim Guay, PMP, CSM, CSD, PMI-ACP, CLSSS
- 2. Who is cPrime? Engaged for Your Project Management Success
- 3. After the webinar… • We will send directions to collect the PDU you will earn from attending this webinar • We will also send a links to the recorded webinar and presentation slides once they are posted online For more information, visit www.cprime.com
- 4. Your Instructor • Tim Guay has over 25 years of IT experience and has applied Agile methodologies since 2002. • Enterprise Data Warehouse Specialist for 6 1/2 years • Managed major DW projects • PMP Certified since 2001, CSM since 2008, PMP-ACP since 2012, and Lean Sensei since 2013. • Clients have included government agencies, start-ups, and Fortune 500 corporations. • Agile trainer and coach. 4
- 5. Agenda • Agile Data is Possible • Why do it? • Guiding Principles • Evolutionary design • Database Refactoring • Hyper-normalization and Generalization • Agile Modeling • Q & A 5
- 6. Agile Data Is Possible • There are many who say that creating an enterprise- level database or data warehouse requires BDUF • It is possible and it is actually the better way to go as both Kimball and Inmon attest • Though Kimball’s architecture is best suited and will be the one that underlies my presentation. • Best because: • Bottom-up approach • Conformed Dimensions and Bus • Matrix Bus 6
- 7. Agile Data is Possible Goals of Agile Data Architecture •To architect to support the delivery of working DW/BI functionality early and continuously to our customers •To architect for change •Scott Ambler key thought-leader in this space 7
- 8. Why Do It? • Agile Myths - Too risky, no planning, no design, no documentation, cowboy coding, only good for small projects • Waterfall realities - Overall failure rate 29% (Standish), DW failure rate 50%+ (Gartner) 8
- 9. Why Do It? • DW Failure Modes : • Insufficient business involvement • Underestimating the complexity and scope • Not anticipating or allowing change • Misunderstood expectations • Overcomplicated architecture • Poor understanding of the data 9
- 10. Guiding Principles Agile Principles 1.Our highest priority is to satisfy the customer through early and continuous delivery of valuable software 2.Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage 3.Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale 4.Business people and developers must work together daily throughout the project 5.Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done 6.The most efficient and effective method of conveying information to and within a development team is face-to-face conversation 10
- 11. Guiding Principles Agile Principles 7.Working software is the primary measure of progress 8.Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely 9.Continuous attention to technical excellence and good design enhances agility 10.Simplicity — the art of maximizing the amount of work not done — is essential 11.The best architectures, requirements, and designs emerge from self- organizing teams 12.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. 11
- 12. Evolutionary Design Key Practices •Close collaboration between DBAs and developers •Each developer gets their own DB instance and test data •Continuous integration into the shared master •Automate the refactoring •Automatically update the developer instances whenever the master is changed •Have a clear DB access layer within the code •Beware of delivering one-off solutions 12
- 13. Evolutionary Design Laying the Foundation - Conformed Dimensions •Conformed dimensions are descriptive master reference data that are referenced in multiple dimensional models •Fundamental to the Kimball approach •Enables Agile DW/BI by levering existing CD’s •Start by identifying a subset of attributes that have significance across the enterprise and iteratively grow from there •Failure to create conformed dimensions from the start will result in significant technical debit and is one of the key reasons for Agile DW project failure 13
- 14. Evolutionary Design Laying the Foundation - Bus Matrix •Each column is a conformed dimension •Separate columns describe other information associated to each business process i.e. Owner, etc. •Each row is a business process •Each dimension is associated to a process by an X in the intersecting cell •Meets the Agile principle of just enough documentation 14
- 15. Evolutionary Design Laying the Foundation - Bus Matrix •Can be done in a matter of days with the right people at the table and a skilled facilitator •Solid understanding of data and processes is required •Collaboration is key •Provides the Agile master plan and list of reusable common dimensions •Focusing on one row at a time reduces risks from overly- ambitious plans and supports the Agile principle of rapid development of valuable software 15
- 16. Evolutionary Design Database Encapsulation Layer •Software architecture should include a database encapsulation layer; aka persistence layer or data layer •Hides the physical details of the DB from the business code •If DB changes only this layer needs to be changed •Consolidates all DB access code in ‘one’ place 16
- 17. Evolutionary Design Database Encapsulation Layer - Variations •Single application, single DB - pretty straight-forward •Multiple-applications, single DB - common when there is a legacy DB •Multiple applications, multiple DB •Implement via direct SQL access, DAOs, Persistence Frameworks, or services 17
- 18. Database Refactoring • Essentially normalization after the fact • Are design improvement changes to the schema that still preserve its behavioral and informational semantics • Includes both structural and functional aspects • Can involve doing three changes together 1. Changing the schema 2. Migrating the data to the new schema 3. Changing the DB access code 18
- 19. Hyper Normalization & Generalization • Hyper-normalization – Beyond 3NF • Data Vault with attributes in satellite tables and foreign keys moved to link tables • Allows changes to data relationship without changing data (hub) tables • Hyper-generalization -all hub data moved to single table and have a table of tables to ID which rows belong to what data category. Also only one link table needed • Reduces complexity and collateral damage from changes 19
- 20. Database Refactoring Examples include: •Apply Standard Types to Similar Data •Consolidate Key Strategy for Entity •Encapsulate Common Structure With View •Introduce Column Constraint •Introduce Common Format •Introduce Lookup Table •Migrate Database Method to Application •Rename Column •Replace One-To-Many With Associative Table •Replace View With Method(s) •Split Column 20
- 21. Database Refactoring Refactoring Enablers: •Regression testing •Strong configuration management •Close collaboration •Just do it! 21
- 22. Agile Modeling • Scott Ambler developed the concept of Agile Modeling • Agile models are just barely good enough • Agile models are developed iteratively • Starts with a light-weight envisioning session to create a domain model. To that I would add developing a Bus Matrix and defining a core set of conformed dimensions • With each iteration develop just barely enough of the data model to support development of the sprint backlog 22
- 23. Questions 23