By Thoughtworks | Building data as a product: The key to unlocking Data Mesh's potential with Harmeet Sokhi & Vishal Srivastava
- 1. © 2022 Thoughtworks | Confidential By Thoughtworks 1
- 2. © 2022 Thoughtworks | Confidential By Thoughtworks Building data as a product: the key to unlocking Data Mesh's potential 2
- 3. © 2022 Thoughtworks | Confidential Harmeet Sokhi 3 Vishal Srivastava ● Rich experience in implementing data platforms. ● Passionate about data mesh ● Member & Contributor of Global Data Mesh guild at Thoughtworks. ● Design and Implement data engineering platforms, data Products and Machine Learning architectures ● Passionate about helping organizations unleash the potential of data thru democratisation ● Co-organizer of the Data Engineering Melbourne meetup
- 4. © 2022 Thoughtworks | Confidential Agenda 4 Story of Essence Financials K’s new use case Data product plot Conclusion Filling the gaps
- 5. © 2022 Thoughtworks | Confidential Essence Financials 5 5 © 2021 Thoughtworks Vision : To keep market position as a leading financial service provider
- 6. © 2022 Thoughtworks | Confidential Significant investment in data Reflection of past Ambitions and Challenges 6 Ambition was to become data oriented organisation Data Teams were not able to keep up with the demand. Data Trustworthiness was questionable Reliance on central data engineering team. Siloed and hyper-specialized ownership
- 7. © 2022 Thoughtworks | Confidential Data Mesh to the rescue Scale Analytical Data 7 Socio-technical Decentralized
- 8. © 2022 Thoughtworks | Confidential 8 8 Why Essence Financials moved away from “data sets”? Data Discoverable Addressable Understandable Trustworthy Natively Accessible Valuable Secure Data Product > Dataset Interoperable
- 9. © 2022 Thoughtworks | Confidential Data Mesh to the rescue Essence Financials got benefited by four interconnected & non-negotiable pillars 9 Dehghani, Z. (2022). Data Mesh: Delivering Data-Driven Value at Scale (1st ed.). O’Reilly Media. Domain-oriented Ownership Data as a Product Self-serve data platform Federated Computational Governance {G} {G} {G}
- 10. © 2022 Thoughtworks | Confidential Logical architecture of “a” Data Product in Essence Financials Transformation Governance Platform Output Port(s) Input Port(s)
- 11. © 2022 Thoughtworks | Confidential Data Product: Categories in Essence Financials 11 Source oriented Data Product Fit-for-purpose Data Product Customer oriented Data Product * * * * * *
- 12. © 2022 Thoughtworks | Confidential Data Product: Typical deployment in Essence Financials Data Product Developers Data Product Code (Business Logic) Data Product Model Spec (Platform & Governance Configuration) Data Product Owner Problem / use case Utility Deploy
- 13. © 2022 Thoughtworks | Confidential Process after adoption 13
- 14. © 2022 Thoughtworks | Confidential 14 14 Tying it back: Designing Data Product 14 © 2022 Thoughtworks Gap analysis Build & Measure Data Strategy Data Mesh is the right architectural choice Use Case Define
- 15. © 2022 Thoughtworks | Confidential K’s new use case 15 /Define Gap analysis Use Case Build & Measure Define
- 16. © 2022 Thoughtworks | Confidential Use Case 16 Promote prime loans
- 17. © 2022 Thoughtworks | Confidential Elevator Pitch FOR Marketing Team WHO Want to sell high value loans THE “Promote Prime Loans” IS A Recommendation Engine THAT will allow them to spend more time in engaging with relevant customers UNLIKE now when then send blanket communication to everyone OUR PRODUCT Resulting in more conversions as well as saving Time spend in Operations 17 K’s new use case PROMOTE PRIME LOANS
- 18. © 2022 Thoughtworks | Confidential Agenda 18 Story of Essence Financials K’s new use case Data product plot Conclusion Filling the gaps
- 19. © 2022 Thoughtworks | Confidential SLO 19 Gap analysis Use Case Build & Measure Define Data Landscape Data Product Map CFRs Define
- 20. © 2022 Thoughtworks | Confidential Data Landscape - SLO-Data Product Map - CFRs 20 Gap analysis Use Case Build & Measure Define
- 21. © 2022 Thoughtworks | Confidential 21 Data Product: Data Landscape Understand Current State,Pains & Aspirations Customer Information Customer transaction History Acquire Productionize & Execute Curate Model
- 22. © 2022 Thoughtworks | Confidential Data Landscape -SLO- Data Product Map - CFRs 22 Gap analysis Use Case Build & Measure Define
- 23. © 2022 Thoughtworks | Confidential 23 How often it is used? How many people or services use it? How complete the data need to be? How often is it updated? When does it need to be updated? How accurate does the data need tobe? How fresh the data need to be? Every second Millions 100% complete Every second 24/7 Perfectly Accurate Realtime Every Hour Thousands Every Hour Within minutes Several times a day Hundreds Mostly complete Several times a day Extended Workday Some room of error Within Hours Daily Dozens Daily Within the day Weekly A couple of teams Can tolerate Missing data Weekly During one part of the day As long as its representa tive Within the week Monthly + Just a few people Monthly + Within the month PROMOTE PRIME LOANS Records with 80% complete data with 100% accuracy are refreshed daily Data Product: SLOs Service Level Objectives assist in determining technical design
- 24. © 2022 Thoughtworks | Confidential Data Landscape - SLO-Data Product Map - CFRs 24 Gap analysis Use Case Build & Measure Define
- 25. © 2022 Thoughtworks | Confidential Data Product: Integrated Data Product Map What could be different types of data products? 25 Sources Source oriented Data Product Fit-for-purpose Data Product Customer Oriented Data Product Operational Reporting/Analytics PROMOTE PRIME LOANS
- 26. © 2022 Thoughtworks | Confidential 26 Sources Source oriented DP Fit-for-purpose Data Product Customer Oriented Data Product Operational Reporting/Analytics PROMOTE PRIME LOANS Salesforce Snowflake Data Product: - Integrated Data Product Map Start from customer oriented data product Domain: Marketing Subdomain: Promotion
- 27. © 2022 Thoughtworks | Confidential 27 Sources Source oriented DP Fit-for-purpose Data Product Customer Oriented Data Product Operational Reporting/Analytics PROMOTE PRIME LOANS Salesforce Snowflake Data Product: Integrated Data Product Map Now lets focus on source oriented data product and their attributes Domain: Marketing Subdomain: Promotion Customer Information CUSTOMER INFO Customer History Table CUSTOMER REPAYMENT HISTORY Domain: Customer Subdomain: Payment
- 28. © 2022 Thoughtworks | Confidential Data Product: Integrated Data Product Map We need intermediate Data Product also CUSTOMER REPAYMENT STATUS Sources Source oriented DP Fit-for-purpose Data Product Customer Oriented Data Product Operational Reporting/Analytics PROMOTE PRIME LOANS Salesforce Snowflake Domain: Marketing Subdomain: Promotion Customer Information CUSTOMER INFO Customer History Table CUSTOMER REPAYMENT HISTORY Domain: Customer Subdomain: Payment Domain: Marketing Subdomain: Promotion
- 29. © 2022 Thoughtworks | Confidential SLO-Data Landscape - Data Product Map - CFRs 29 Gap analysis Use Case Build & Measure Define
- 30. © 2022 Thoughtworks | Confidential 30 30 Data Product: Cross Functional Requirements (CFRs) 30 © 2021 Thoughtworks Data Access Policy Data standardisation Rules Data Classification Rules Data retention policy TFN
- 31. © 2022 Thoughtworks | Confidential SLO-Data Landscape - Data Product Map - CFRs 31 Gap analysis Use Case Build & Measure Define
- 32. © 2022 Thoughtworks | Confidential Agenda 32 Story of Essence Financials M’s new use case Data product plot Conclusion Filling the gaps
- 33. © 2022 Thoughtworks | Confidential Architecture Gap analysis 33 Gap analysis Use Case Build & Measure Define
- 34. © 2022 Thoughtworks | Confidential Architecture Gap Analysis 34 Data Product Architecture Governance Platform Capabilities Gap analysis Use Case Build & Measure ( Define
- 35. © 2022 Thoughtworks | Confidential Orchestration Managed Airflow 35 Data Product: Gap Analysis Data storage Snowflake Marketing data mart PROMOTE PRIME LOANS Governance Transformation Data product code Platform Capabilities Snowflake connector Salesforce connector Snowflake connector
- 36. © 2022 Thoughtworks | Confidential Data Product- Promote Prime Loan CUSTOMER REPAYMENT STATUS PROMOTE PRIME LOANS Salesforce Snowflake Customer Information CUSTOMER INFO Customer transaction history CUSTOMER REPAYMENT HISTORY Platform Capabilities Governance Managed Airflow Snowflake ci/cd Observability
- 37. © 2022 Thoughtworks | Confidential Agenda 37 Story of Essence Financials M’s new use case Data product plot Conclusion Filling the gaps
- 38. © 2022 Thoughtworks | Confidential /Build & Measure 38 Gap analysis Use Case Build & Measure Define
- 39. © 2022 Thoughtworks | Confidential Build & Measure 39 Gap analysis Use Case Build & Measure ( Define Learn Build Measure
- 40. © 2022 Thoughtworks | Confidential Build & Measure 40 Gap analysis Use Case Build & Measure ( Define Data Product Score Rule based User feedback KPIs SLOs
- 41. © 2022 Thoughtworks | Confidential Happy K 41
- 42. © 2022 Thoughtworks | Confidential Recap 42 Gap analysis Define Data Landscape - SLO Data Product Map - CFRs Use Case Build & Measure Data Product Architecture Governance Platform Capabilities Learn Build Measure
- 43. © 2022 Thoughtworks | Confidential “A Journey of Thousand Miles start with a single step” -Lau Tze 43
- 44. © 2022 Thoughtworks | Confidential We look forward to working with you Harmeet Kaur Sokhi Lead Data Consultant harmeet.sokhi@thoughtworks.com Vishal Srivastava Senior Data Consultant vishal.srivastava@thoughtworks.com 44 Please share your feedback