Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
4 likes1,891 views
The document presents an overview of establishing a DevOps practice specifically for analytics, emphasizing the integration of development and operations teams to improve coordination in data science and data analytics. It highlights the growing need for speed and efficiency in analytics processes, facilitated by cloud platforms and automation, and outlines best practices for creating a successful DevOps culture within an organization. Key aspects include the importance of teamwork, continuous integration, and automation, along with the unique challenges posed by analytics projects compared to traditional software development.
Creating a DevOps Practice for Analytics -- Strata Data, September 28, 2017
- 1. Strata 2017 Creating a DevOps Practice for Analytics Bob Eilbacher September 28, 2017
- 2. Agenda About Caserta DevOps DevOps for Analytics Organization and Teams Questions
- 3. About Caserta Data Intelligence Consulting and Modern Data Engineering Award-winning data innovation Internationally recognized work force Strategy, Architecture, Governance, Implementation
- 4. About Caserta Architecture & Design Implementation Services Disruption Management Strategic Technical Consulting Training & Education Application Innovation Cloud Management
- 5. What is DevOps for Analytics? First some terminology… DevOps Associated with movement primarily in application development space for last 5-10 years Focused on very fast and continuous software product releases Think intra-day Prod releases at Netflix, Amazon, etc. Convergence of development and operations methodologies to minimize TTR Tons of resources – devops.com, DZone
- 6. What is DevOps for Analytics? Some more terminology… DataOps Re-emergent term Seems to have a broader context Applying DevOps to data management or to handling backend databases Also tends to carry real legacy connotation Manual operations of database backups and restores,
- 7. What is DevOps for Analytics? And finally… AnalyticsOps This is a term that we see starting to be used more Its focused on applying DevOps practices within a data analytics and data science context This is the area we’re interested in for this talk We’ll use the terms AnalyticsOps or the more explicit DevOps for Analytics interchangeably
- 8. DevOps… Speak with anyone and they will tell you first that DevOps is a culture Based primarily on teamwork
- 9. DevOps…
- 10. DevOps… Speak with anyone and they will tell you first that DevOps is a “culture” Based primarily on teamwork Aims to address the underlying conflict between development and operations objectives Innovation @ speed vs. Performance @ quality Change vs. Stability Culture is not “implemented” It needs to evolve Good news is it can be seeded
- 11. DevOps… It works! 75% of IT and product dev organizations were successfully using DevOps to some extent – Source: RightScale 2016 State of the Cloud Report It’s flexible No two companies’ DevOps approaches will look the same Infinite number of ways to create teamwork A reflection of the organization itself
- 12. DevOps… DevOps tenets Continuous Integration Test Automation Continuous Delivery Continuous Deployment End-to-end automation is still aspirational for most companies Justify how much automation you need based on business requirements.
- 13. DevOps… What DevOps is not is a toolchain implementation Tools help the team execute within the culture Don’t run out and put an end to end chain in place and then expect adoption Lets talk about tools for a minute … Explosion of both open-source and commercial DevOps tooling Serve every discrete need requirements management, SCM, test automation, defect tracking, build, deployment, monitoring and more 1,500+ tools available
- 14. DevOps… Tooling categories: Code : Code development, version control tools, code merging Build : Continuous integration tools, build status Test : Test and results determine performance Package : Artifact repository, application pre-deployment staging Release : Change management, release automation Configure : Infrastructure configuration and management, Infrastructure as Code tools Monitor : Applications performance monitoring, end user experience
- 16. Why DevOps for Analytics? “The fact is that analytic teams are being compared by their businesses to Amazon Prime – 2-day delivery of almost anything” Source: Unknown
- 17. Why DevOps for Analytics?
- 18. Why DevOps for Analytics? A couple of recent real world examples… Data Science Rock Star Process Overengineering
- 19. Why DevOps for Analytics? Analytics and data science projects, what used to take months to achieve is happening in days or hours Businesses typically like that and want more… Enabled by the strong trend toward cloud analytic platforms/services Infrastructure as code (IaC) allows extension of software development practices to servers and infrastructure We can automate the build of complex analytic pipelines - storage, processing engines, etc. with relative ease
- 20. DevOps for Analytics DevOps for Analytics combines the development and operations teams and establishes best practices that improve coordination between data science and operations BUT… Data Science and Analytics are different from application development Especially in a Big Data environments - need big data to test big data applications Much more diverse mix of tools and technologies – not just java Some differences in approach are needed
- 21. DevOps for Analytics AnalyticsOps this is still in its early days There aren’t any real solid industry success stories published People are still trying to figure out what works and aren’t’ open kimono and sharing experiences just yet Not a lot of experienced practitioners But there are some early themes and guidelines emerging
- 22. DevOps for Analytics Environments Separate DEV and PROD environments Should you reuse any of the PROD data assets? Separate landing area, destination area (Data Lake), etc. Trickier with increasing data volumes – do it smart to avoid double costs Sharing compute cluster resources is OK Make all job inputs and outputs configuration driven (PROD and DEV code doesn’t change) – for CI
- 23. DevOps for Analytics Automated Testing It’s almost impossible to get full code coverage How do you unit test SPARK SQL scripts? Regression tests? Data validation? Test data is a complex problem – handle as a cross-functional initiative. Analytic results are often buried in complex outputs, QA becomes forensic data analysis Automate what you can, supplement with community based real-world data testing in a parallel Dev/Test environment The role of the Test/QA Engineer is still really important Test/QA Engineers need Data Engineering experience
- 24. DevOps for Analytics Monitoring Tracking and analyzing intra-day demand and longer term trends in infrastructure performance (standard DevOps) But then… By their nature analytics processes require monitoring and tuning over time with real-world inputs Data drifts; Predictive models have a finite lifetime Silent failures Feedback to developers so they can see how their code is performing and affecting the Prod environment Continuous improvement The next wave is analytics on analytics…
- 25. DevOps for Analytics Emerging DevOps for Analytics environment usually contain SCM CI Repo to store analytics app Repo to store configuration An API to deploy to the cluster Mechanism to monitor behavior and performance
- 26. DevOps for Analytics Organization Building a DevOps for Analytics culture is not an easy undertaking Should fall under the purview of a dedicated data organization These organizations are typically lead by the Chief Data Officer More recently by Chief Data Scientist a Chief Analytics Officer Key responsibilities include Fostering adoption Clarifying and aligning to the business' vision Securing reasonable funding
- 27. DevOps for Analytics Organization The goal over time is to create lean, highly performant, cross- functional, extremely effective teams Business Stakeholders Data Engineers Data Analysts & Data Scientists QA Operations All of these skills are important - but when in doubt get more Data Engineers! Everyone on team has an equal voice Everyone codes & Everyone needs to know what Prod looks like
- 28. DevOps for Analytics Organization Start-up Condition: Bring in an experienced set of DevOps for Analytics Engineers Help define the culture, lead by example Identify the Innovators and get them involved and leading The DevOps Engineers job is to ultimately engineer themselves out of the equation Source: Matthew Skelton, DevOps Patterns - Team Topologies
- 29. Final Thoughts “We aim to engineer systems and processes to better integrate development and operations, resulting in decreased time to market and an application infrastructure that is instrumented, scalable and fault tolerant… and immortal!” - Will Liu, Equinox Data Team
- 30. Final Thoughts There are plenty of benefits in establishing a DevOps for Analytics culture for your organization For the business: Speed to insight For the teams: Professional and personal satisfaction Be Fearless – go build your own DevOps for Analytics culture!
- 31. Questions
- 32. Happy Birthday Joe Caserta!
- 33. Thank You Bob Eilbacher Vice President Operations, Caserta bob@casertaconcepts.com Upcoming Training Opportunity: Caserta is hosting 3 Days of Training Courses October 18-20th in NYC, taught by Joe Caserta, co-author of The Data Warehouse ETL Toolkit: Day 1: Agile Data Warehouse Design & Dimensional Modeling Day 2: ETL Architecture & Design Day 3: Big Data for Data Warehouse Practitioners More info at casertaconcepts.com/event/