Decision-Centric Dashboards with DMN at Two Fortune 200 Financial Companies
5 likes2,409 views
The document discusses the development and implementation of decision-centric dashboards using Decision Model Notation (DMN) at two Fortune 200 financial companies, focusing on improving organizational decision-making processes. It highlights the lack of structured methods for dashboard design and the importance of integrating technology to support user-specific decision requirements. Case studies illustrate the transition from traditional metrics-focused dashboards to a more user-driven, decision-support framework that enhances usability and engages stakeholders effectively.
![[A] Process: Operations
Resource Assignments
QA Test Execution
Defect Management
Time to Release Date
QA Plan Exit Criteria
[B] Process: Project Management
Project Initiation Sync
Project Phase Sign-Off
Performance, Security, Accessibility Sign-Off
Project Transition
QA Operation Blocks
[C] Process: Effectiveness
Requirements Traceability
Process Standards
Test Plan Gaps
Production Stability
Defects Root Cause Analysis
[D] Process: Strategy
Cost Control
Schedule Management
Automation Metrics
Environment Stability
Reuse Metrics
Agile Support
Defect Linkage
Defect Status
Defect Withdraw (rejected) Rate
Defect Reopen Rate (from closed)
Defect Retest Fail Rate
Defects Severity
Defects Priority
Defect List
Defect by Tester
Tester Efficiency
Defect Velocity Detection
Defect Velocity Closure
Defect Density
Outstanding Defects
Retargeted Defects
Transitioned Defects
Root Cause Analysis
# of Runs
# Executed Steps
Test Velocity
Defect Effort
Testing Efficiency
Execution Trend
Planned vs. Actual
First Pass Rate
Not Completed Analysis
Test Cases Created
Test Cases Modified
Release Start Date
Release End Date
Days Left in the Release
Days Left in Test Cycle
% of Requirements Without Test Cases
% of Test Cases Without Requirements
Requirement Traceability
Manual vs. Automated
Reuse Test Scenarios vs. New Test Scenarios
INFORMATION SOURCES: RATIONALIZED
25
DATA](https://image.slidesharecdn.com/bbc14dashboardsusingdmn3-150112111559-conversion-gate02/85/Decision-Centric-Dashboards-with-DMN-at-Two-Fortune-200-Financial-Companies-25-320.jpg)
Decision-Centric Dashboards with DMN at Two Fortune 200 Financial Companies
- 1. DECISION-CENTRIC DASHBOARDS WITH DMN AT TWO FORTUNE 200 FINANCIAL COMPANIES Gagan Saxena, VP Consulting, Decision Management Solutions Andrea Gonsalves, Director QA, TIAA-CREF Financial Services Shir Goldberg, VP Business Development, Assure Corporation
- 2. q There is no structured method for designing and implementing Dashboards q Wide variety of implementations q Dashboards are built around individual preferences of select ‘super users’ q Constant changes as new users arrive WHY ARE WE HERE? WHAT ARE WE SOLVING? 2 q Go back to first principles q Dashboards are for Decision Support. So, start with Decisions. Organizational Decisions q Use an industry standard for modeling Decisions q Decision Model Notation (DMN) from Object Management Group (OMG) q Use flexible tech platform with adaptors q TotalView from Assure – Leader in Application Lifecycle Management (ALM) Analytics SOLUTION
- 3. 1. Introduction - Dashboards support Decisions 2. Decision Model Notation (DMN) and Decision Models 3. Case Study 1 – Requirements & Design 4. Case Study 2 – Implementation 5. Takeaways OUTLINE 3
- 4. 1. INTRODUCTION DASHBOARDS SUPPORT DECISIONS
- 5. ABOUT DECISIONS 5 Dashboards Support Tactical and Operational Decisions
- 7. DATA VISUALIZATION ≠ DASHBOARD DESIGN 7 Data Visualization is a Pattern Discovery Technique; Not part of a Dashboard
- 8. DASHBOARD IMPLEMENTATION METHODOLOGY 8 Business Requirements are best captured with formal Decision Modeling
- 9. DASHBOARDS THAT PREDICT & RECOMMEND 9
- 10. NEED TO INTEGRATE TECHNOLOGIES 10
- 11. 2. DECISION MODEL NOTATION (DMN) AND DECISION MODELING
- 12. DECISION MODEL AND NOTATION (DMN) 12 INDUSTRY STANDARD FROM OBJECT MANAGEMENT GROUP January 2014 Provides the constructs that are needed to model decisions, so that Organizational decision-making can be readily depicted in diagrams, accurately defined by Business analysts, and (optionally) automated.
- 13. BASIC ELEMENTS OF A DECISION MODEL 13 Decisions require Information and Knowledge
- 14. USING A DECISION REQUIREMENTS MODEL 14
- 15. AUTOMATION BOUNDARIES ! IMPLEMENTATION 15
- 16. REF ARCHITECTURE (DECISION REPOSITORY) 16
- 17. DECISION MODEL DRIVES DESIGN 17
- 18. 3. CASE STUDY ONE: DASHBOARD REQUIREMENTS AND DESIGN
- 19. u Needed Enterprise IT Management Dashboards u Manage 100+ projects being released every 2 weeks into production u Across hundreds of critical applications u Mainframe, client-server, web and mobile applications. u Consolidate multiple Dashboards and Reports from u Requirements Management u Quality Management u Defect Management u Project Management u Cost Management FORTUNE 200 FINANCIAL COMPANY
- 20. SOLUTION FOR A CONSOLIDATED DASHBOARD 20 Flexible Dashboards and Adaptor Technology (TotalView from Assure) Decision Requirements Modeling Data Discovery and Gap Analysis Decision Centric User Interface
- 21. u Requirement Gathering u Workshops with Stakeholders u Existing Reports u Identify key focus areas u Identify Decisions u Metrics Exploration DECISION MODELING FORMALIZED ANALYSIS 21 Decision Modeling u Data Analysis u Identify Data Sources u Identify Data Attributes u Identify Data Gaps u Identify Joining Information
- 22. DATA ACTIONS/ PROCESSES DECISIONS INFORMATION/ KNOWLEDGE STRATEGY DASHBOARDS 1 2 3 45 DASHBOARD DESIGN FOR DECISION SUPPORT 22
- 23. A. Operations: Make Release Date B. Project Management: Sync Project Initiation, Delivery & Close C. Effectiveness: Ensure 100% Quality D. Strategy Control: Meet Budget & Strategic Goals Decision Models embed Strategy into Processes 1. Lower QA Costs 2. Increase Test Automation 3. Transition to Agile Delivery 4. Standardization of Tools & Processes 5. Uniform & Transparent Quality Metrics 6. Increase Test Assets Reuse LINK STRATEGY TO PROCESSES 23 STRATEGY 1 DECISIONS 2 ACTIONS/ PROCESSES 3
- 24. QA Operations: Make Release Date 24 PROCESS: GOAL
- 25. [A] Process: Operations Resource Assignments QA Test Execution Defect Management Time to Release Date QA Plan Exit Criteria [B] Process: Project Management Project Initiation Sync Project Phase Sign-Off Performance, Security, Accessibility Sign-Off Project Transition QA Operation Blocks [C] Process: Effectiveness Requirements Traceability Process Standards Test Plan Gaps Production Stability Defects Root Cause Analysis [D] Process: Strategy Cost Control Schedule Management Automation Metrics Environment Stability Reuse Metrics Agile Support Defect Linkage Defect Status Defect Withdraw (rejected) Rate Defect Reopen Rate (from closed) Defect Retest Fail Rate Defects Severity Defects Priority Defect List Defect by Tester Tester Efficiency Defect Velocity Detection Defect Velocity Closure Defect Density Outstanding Defects Retargeted Defects Transitioned Defects Root Cause Analysis # of Runs # Executed Steps Test Velocity Defect Effort Testing Efficiency Execution Trend Planned vs. Actual First Pass Rate Not Completed Analysis Test Cases Created Test Cases Modified Release Start Date Release End Date Days Left in the Release Days Left in Test Cycle % of Requirements Without Test Cases % of Test Cases Without Requirements Requirement Traceability Manual vs. Automated Reuse Test Scenarios vs. New Test Scenarios INFORMATION SOURCES: RATIONALIZED 25 DATA
- 26. DATA GAPS: IDENTIFIED 26 Source Gaps Plan 1 HP- ALM Linking Releases Linking Cycles Using Defects Using Requirements Excel Data Upload Frequency Excel Data Upload by a Single Person ALM Used for System Test Only Process Changes/ Training 2 CMDB EPMS Fields Not Available DEV 3 CMDB HR Fields Not Available DEV 4 CMDB QA PLAN Service Not Available DEV 5 CMDB RFC NONE None
- 27. u Four Tabs for the four levels of Decisions u Operations, Project Management, Effectiveness, Strategy u Two organization dimensions for filtering u Hub and Unit u Two dimensions for Time u Release and Regular Calendar u “Details” Panel u Displays details of Selection u “Summary” Panel u Traffic Lights & Numbers u “Time to Release” Panel repeated u Operations & Project Mgt u “Information” button can link to description of Decision Model u E.g. on Operations Summary Panel DESIGN PRINCIPLES & NOTES 27 = ALM Data Defect Retest Fail Rate Defects Severity Defects Priority Defect List Defect by Tester Tester Efficiency Defect Velocity Detection All ALM User Assignments = Possible ALM QA Test Lead = Known Ext Data EPMS# with no QA Estimates = Unknown Ext Data
- 28. IT SHARED SERVICES – QUALITY MANAGEMENT DATA AS OF: MAR 14, 2014 10:27a CT OPERATIONS PROJECT MGT EFFECTIVEN ESS STRATEGY SUMMARY TIME TO RELEASE TEST EXECUTIION RESOURCE ASSIGNMENTS DEFECTS BLOCKING ISSUES UNIT HUB MARCH 2014 RELEASE # of Runs # Executed Steps Test Velocity Defect Effort Testing Efficiency Execution Trend Planned vs. Actual First Pass Rate Not Completed Analysis Test Cases Created Test Cases Modified Defect Linkage Defect Status Defect Withdraw (rejected) RateDefects Severity Defects Priority Defect List Defect by Tester Tester Efficiency Defect Velocity Detection Defect Velocity Closure Defect Density Release Start Date Release End Date Days Left in the Release Days Left in Test Cycle Prioritized Defects by Tests Blocked QA Test Lead All ALM User Assignments DETAILS 28
- 29. 4. CASE STUDY TWO: IMPLEMENTATION AT TIAA-CREF
- 30. TIAA-CREF FOUNDED ALMOST 100 YEARS AGO 30 From Teacher Retirement Services to Full-Service Financial Services
- 31. THE QUALITY ASSURANCE TEAM 31 BusinessIntelligence/ Communication RITQAServiceOfferings RIT QA – 2014 Themes Cutting Edge Delivery Optimized Delivery Quality High-Touch QA Services Systems Integration Testing (SIT) From a Mom & Pop Shop to a Fortune 100 Company Quality Assurance Assess and Measure SDLC Quality Test Data Mining Test Program Management Freeze Exception Management
- 32. TIAA’S TRANSFORMATION PLAN 32 ü Predictability ü Systema;c Approach ü Improved Delivery Quality ü Ac;vely Monitor Against Plan ü Use of Metrics and Predic;ve analysis ü Manage Risk X ‘Trust me, I’ve got it covered’ X Reac;ve X Dependent on Individuals X Status Quo X Not Scalable Using Business Intelligence
- 33. DASHBOARD IMPLEMENTATION 33 Phase1 Dedicated Metrics Expert But not engaged in Projects Ended up with siloed views of Test execution & Defects Focused only on needs of QA (one view/one role) Phase2 Started with a prototype focused on usability and simplification Interviewed actual users Examined Decisions Examined Roles Training and Education; socialized usage for key decisions Phase3 Enhanced Views with formal decision modeling Focused on interpreting data and recommending actions Utilize advanced analytics A Dashboard that few people used A powerful Dashboard, broadly used (Expected) A Prescriptive Dashboard
- 34. DECISION CENTRIC TECHNIQUES 34 User Goals and Actions that User-Roles takes to achieve them Decisions that User-Roles make while taking those Actions Information (Data) and Knowledge (Calculated Analytics) needed Dashboard Designs with User-Role specific Decisions grouped together 1 2 3 4
- 36. RESULTS 36 0 100 200 300 Dec 2013 Jan 2014 Feb 2014 Mar 2014 Apr 2014 May 2014 TotalView Unique User Sessions Rapid Adoption across the Business Relevant Metrics for clear Decision-Making No ‘clutter’ of pretty but extraneous information One Dashboard – One Version of Truth Enabled Shift Left: Right Information at the Right Time for the Right Decision 2.5 X increase in user sessions and doubled user base to 530 users
- 37. 5. TAKEAWAYS
- 38. u Dashboards support Decisions u Identify Decisions First u Get Decision Requirements and Design around them u Decision Model Notation (DMN) is a formal standard. Use it. u Data Visualization and Data Discovery interfaces u Are NOT Dashboards u Are useful for Pattern Discovery and related Insights TAKEAWAYS 38
- 39. BACKUP SLIDES
- 40. 40 QA Project Mgt: Sync Initiation, Delivery, Close PROCESS: GOAL
- 41. 41 QA Effectiveness: Ensure 100% Quality PROCESS: GOAL
- 42. 42 QA Strategy Control: Meet Budget & Strategy PROCESS: GOAL
- 43. PROJECT MGT OPERATION S EFFECTIVEN ESS STRATEGY SUMMARY TIME TO RELEASE INITIATION DETAILS DELIVERY CONTROL BLOCKED OPERATIONS TRANSITION IT SHARED SERVICES – QUALITY MANAGEMENT DATA AS OF: MAR 14, 2014 10:27a CT UNIT HUB MARCH 2014 RELEASE Outstanding Defects Retargeted Defects Transitioned Defects “Test Data Not Available” w Status “Test System Outages” w Status EPMS# with no ALM# EPMS# with no QA Estimates EPMS# with no QA Lead Assigned EPMS# with no QA Plan Shell Project Phase Sign-offs Status With dates “Test Resources Not Available” w Status Release Start Date Release End Date Days Left in the Release Days Left in Test Cycle Performance Test = Y/N/na Security Test = Y/N/na Accessibility Test = Y/N/na 43
- 44. EFFECTIVENE SS OPERATION S PROJECT MGT STRATEGY SUMMARY PRODUCTION DEFECTS REQUIREMENTS COVERAGE DETAILS BEST PRACTICES TEST PLAN GAPS ROOT-CAUSE ANALYSIS IT SHARED SERVICES – QUALITY MANAGEMENT DATA AS OF: MAR 14, 2014 10:27a CT UNIT HUB MARCH 2014 RELEASE % of Reqts. Without Test Cases % of Test Cases Without Reqts. Requirement Traceability Root Cause Analysis Prod Defects by Application Prod Defects per 10k FP Prod Defects by Priority Prod Defects Pain (min) # of Test Cases # of Steps in Test Cases Test Cases Without Defects Test Cases Never Failed Test Cases That Always pass Test Cases without Steps Test Cases Planned per Day Test Case # of Runs 44
- 45. STRATEGYOPERATION S PROJECT MGT EFFECTIVEN ESS SUMMARY RESOURCING & COSTS QA ENVIRONMENT STABILITY DETAILS ASSET REUSE AUTOMATION AGILE DELIVERY IT SHARED SERVICES – QUALITY MANAGEMENT DATA AS OF: MAR 14, 2014 10:27a CT UNIT HUB MARCH 2014 RELEASE Manual vs. AutomatedReuse Test Scenarios vs. New ST Automation % RR Automation% Late Performance Engagement Late Data Engagement QA Blockage Trends Project Approvals Trends Project Change (BCR) Trends Agile v Waterfall Project Stats Sonar- Complexity Sonar- Coverage Test Scenarios checked into Domain Repository QA Resources assigned to Agile Development Roles QA Hours per Project FTE v 3rdParty Utilization/ Bench SCHEDULE PERFORMANCE Average Time in System Test Time to Market 45