My code, my environment, and yes, my data
1 like378 views
This document discusses democratizing data and including it as part of the DevOps toolchain. It argues that data should be made available as a service and provisioned in a secure and compliant manner to empower developers. The document recommends using a DataOps approach and platform like Delphix to virtualize data from various sources and provision production-like test data for developers in an automated way. This helps overcome issues like long test data provisioning times and lack of access to production data, improving the delivery pipeline. Case studies of insurance and banking clients adopting this approach are also presented.
My code, my environment, and yes, my data
- 1. My Code, My Environment, and Yes, My Data Empowering the Developer with Data Sanjeev Sharma VP, Data Modernization
- 2. © 2019 All Rights Reserved. 2 $ whoami • 20+ Years in Software Development and Delivery • Past: IBM Distinguished Engineer and CTO for DevOps Adoption (2013 – 2018) • Author of two DevOps books: • DevOps For Dummies: https://ibm.biz/BdsPMX • The DevOps Adoption Playbook: http://amzn.to/2hH7rt2 • Blog: https://sdarchitect.blog • @sd_architect
- 3. © 2019 All Rights Reserved. 3 1874 - Solid Wood 1947 - Laminated Wood 1968 – Steel Racquet 1993 – Graphite Racquet 2008 – Aerodynamic Racquet Innovation drives the Democratization of Technology
- 4. © 2019 All Rights Reserved. 4 DevOps: Democratizing the CI/CD Pipeline • Democratize the Capabilities across the CI/CD pipeline as services consumable by Practitioners • ‘Shift-Left’ availability of capabilities across delivery pipeline • Raise the abstraction of the role of Specialists to deliver their capabilities as services • Enable better collaboration and communication across functional areas • More effective controls and governance Continuous Integration Continuous Delivery Shift Left Test Shift Left Ops Culture Development SCM Build Package Repo Deploy Testing Staging Production FeedbackPlanning Manage
- 5. © 2019 All Rights Reserved. 5 Automate Infrastructure Automate Data Automate SDLC Innovate 5 Democratize Data Innovate Democratize SDLC Democratize Infrastructure Building a CI/CD Pipeline
- 6. © 2019 All Rights Reserved. 6 6 Democratizing Data DATA
- 7. © 2019 All Rights Reserved. 7 7 Democratizing Data with Delphix
- 8. © 2019 All Rights Reserved. 8 8 Democratizing Data with Delphix Become Data Source Agnostic Make Data Available Self Service Manage Data Like Code Mitigate Data Privacy & Compliance Risks Include Data Management in the DevOps toolchain
- 10. © 2019 All Rights Reserved. 10 Overcoming Cultural Inertia 1. Organizations are not monolithic or homogeneous, with disparate processes, technology stacks, org structures and culture 2. Teams lack trust and collaboration across Functional Siloes. They are not organized for Shared Responsibility 3. Governance Overhead and Manual Approval Processes are major impediments to Agility and Speed DevOps Works fine in co-located ‘2-pizza’ Teams, but not at Enterprise Scale
- 11. © 2019 All Rights Reserved. 11 If you are doing 2- week Sprints, but it takes 3-weeks to get Test Datasets, how long are your Sprints? Democratizing Test Data
- 12. © 2019 All Rights Reserved. 12 It is not possible to patch the software of a missile AFTER it has been launched Quality is dependent on testing with Production-like Data
- 13. © 2019 All Rights Reserved. 13 Overcoming Cultural Inertia • Deliver Data as a Service • Include DBAs in your Squads and Tribes • Make Data Accessible and Agile, in a secure and compliant manner Everyone is responsible for delivering Business Value to Users
- 14. Democratizing Data in the CI/CD pipeline with DataOps
- 15. © 2019 All Rights Reserved. 15 Barriers to Delivering Enterprise Data with Speed and Scale 15 • Massive and continuously growing data volumes • Provisioning cost and delays • Growing data risk management and regulatory compliance burden DATA HAS INERTIA • Resides in heterogeneous sources • Scattered across on- premises and public/hybrid clouds • Managed through custom, complex and/or manual workflows DATA IS COMPLEX • Locked in packaged apps or zero-touch production systems • Incompatible toolchains and workflows • Organizational silos and cultural inertia DATA IS SILOED
- 16. © 2019 All Rights Reserved. 16 A collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and data consumers across an organization. - Gartner DataOps
- 17. © 2019 All Rights Reserved. 17 DataOps for Secure and Compliant Test Data via Self Service 17 Applications Files Databases DATA POD On-Prem Cloud CONNECT APIs VIRTUALIZE Sync Compress Provision SECURE Compliance Policy Masking MANAGE Distribute Audit & Report Manage D B A Developer QA Analyst Data Scientist On-Prem Any Server Private Cloud Public Cloud
- 18. © 2019 All Rights Reserved. 18 Delphix DataOps Platform plugs the toolchain gap Design Develop Test Deploy Source Code Management Continuous Integration Environment Configuration Release Automation Test Automation Test Data Management DistributeMask
- 19. © 2019 All Rights Reserved. 19 INTEGRATION TEST SYSTEM TEST PRODUCTION CODE Virtual DB CODE Virtual DB CODE Virtual DB Production CODE DEV DEV DEV Secure Data Deployment Code and Infrastructure Deployment Provision/Refresh Data Sync and Mask … Trigger BuildPush Delphix Plugin Secure Test Data Management in the Delivery Pipeline 19 Provision/Refresh Data
- 20. © 2019 All Rights Reserved. 20 DataOps for Enterprise DevOps Adoption INTEGRATION TEST SYSTEM TEST
- 21. © 2019 All Rights Reserved. 21 DataOps Adoption Scenarios • Insurance Client: – Had 3 Test environments that were shared by 35 project teams. The three environments were hosted on static infrastructure with limited compute and storage – Went to 35 test environments (one for each team) with Virtualized Data provisioned by Delphix, hosted on the same static infrastructure • Global Bank: – Development teams had dynamic Test Environments. However, they has limited access. Production-like Test Data was not available to Developers. Database refresh took 6-7 days per request – Today all Databases (50+) are provisioned via Delphix into 3,000+ Dev and Test environments. Every development stream (Git branch) has its own Database – Data provisioning governed via automated policy based ‘guardrails’ • Hotel Chain: – Data was not a part of DevOps toolchain. Database provisioning was done manually and only subsets of data was made available to Developers and Testers – Data is a core service of the DevOps toolchain. Database provisioning is part of the Jenkins pipeline and fully available to developers and testers via self-service
- 22. Value Stream Mapping the CI/CD Pipeline
- 23. © 2019 All Rights Reserved. 23 Making the Delivery Pipeline Lean Idea/Feature/Bug Fix/ Enhancement Production Development Build QA SIT UAT Prod PMO Requirements/ Analyst Developer CustomersLine of Business Build Engineer QA Team Integration Tester User/Tester Operations Artifact Repository Deployment Engineer Release Management Code Repository Deploy Get Feedback Infrastructure as Code/ Cloud Patterns Feedback Customer or Customer Surrogate Data Tasks Artifacts Value Stream Mapping to Identify: • Waste • Wait • Rework Video : DevOps Value Stream Mapping
- 24. © 2019 All Rights Reserved. 24 24 • Review the current state o Business goals, IT goals, current initiatives o Requirements o Environments o Repositories o Data Sources /Data Architecture o Roles / Organization o Metrics o Other • Prioritize Waste, Wait states and Rework • Create a first pass at a roadmap to address inefficiencies The whiteboard Next Step: DevOps Value Stream Mapping Workshop