Microservices Approaches for Continuous Data Integration
2 likes1,617 views
The document discusses the evolving landscape of microservices and continuous data integration, highlighting their significance in digital transformation. It emphasizes the need for agile development practices, intelligent systems of engagement, and efficient data integration pipelines to enhance business operations. Additionally, it showcases the role of technologies like Spring Cloud Data Flow in building real-time data processing solutions within modern architectures.
Microservices Approaches for Continuous Data Integration
- 1. 1 Microservices & Continuous Data Integration Dormain Drewitz - Pivotal Matt Aslett - 451 Research Jürgen Leschner - Pivotal
- 2. 2 Today’s speakers Panelist Matt Aslett Research Director 451 Group Panelist Jürgen Leschner Product Team Pivotal Moderator Dormain Drewitz Platform Ecosystem Pivotal
- 3. 3 Agenda 1. Backdrop: Digital Transformation 2. Data Integration: What’s changed? 3. The Case for Microservices for Continuous Data Integration 4. Day 2: People and Platforms 5. Looking toward the Future
- 4. Copyright (C) 2017 451 Research LLC 451 Research is a leading IT research & advisory company Founded in 2000 300+ employees, including over 120 analysts 2,000+ clients: Technology & Service providers, corporate advisory, finance, professional services, and IT decision makers 50,000+ IT professionals, business users and consumers in our research community Over 52 million data points published each quarter and 4,500+ reports published each year 3,000+ technology & service providers under coverage 451 Research and its sister company, Uptime Institute, are the two divisions of The 451 Group Headquartered in New York City, with offices in London, Boston, San Francisco, Washington DC, Mexico, Costa Rica, Brazil, Spain, UAE, Russia, Taiwan, Singapore and Malaysia Research & Data Advisory Events Go 2 Market
- 5. 5 Pivotal Labs Agile Practice Pivotal Data Suite Greenplum GemFire HDB Pivotal Tracker Organize and Plan Spring Build reliably on Spring Boot Concourse Continuous Integration and Deployment Pivotal Cloud Foundry Multi-Cloud Platform for Cloud-Native Apps
- 7. Copyright (C) 2017 451 Research LLC Digital Transformation: What do we mean? 7 Copyright (C) 2017 451 Research LLC When IT innovation is aligned with and driven by a well-planned business strategy to: ! transform how organizations serve customers, employees, and partners ! support continuous improvement in business operations ! disrupt existing businesses and markets ! invent new businesses and business models
- 8. The Case for Microservices Why Microservices? 1. Increased development velocity 2. Organizational scale 3. Increased agility 4. Faster onboarding 5. Cost efficient scaling 6. Attracting/retaining talent 7. Fault isolation Continuous Delivery is a precursor to deliver microservices in a safe manner - Cassandra Shum, Thoughtworks What do Microservices need? 1. Agile development 2. Test-driven development 3. Continuous integration 4. Continuous deployment 5. DevOps 6. Distributed Security 7. Intelligent Traffic Management 8. Visibility
- 10. Copyright (C) 2017 451 Research LLC Engagement and Intelligence: Retail example Customer service is driven by systems of engagement. Traditionally that engagement was physical, and person-to-person Help, advice, suggestions and recommendations were provided and questions were answered (or not) by the employee Even the transaction was performed physically, before being entered into the system of record (financial/ERP, CRM applications) That data was then made available for analysis
- 11. Copyright (C) 2017 451 Research LLC SYSTEMS OF ENGAGEMENT Traditional systems of engagement and analysis SYSTEMS OF ANALYSIS DATA ANALYSTS DECISION MAKERS IT PROS SYSTEMS OF RECORD
- 12. Copyright (C) 2017 451 Research LLC SYSTEMS OF ENGAGEMENT New systems of engagement and analysis SYSTEMS OF ANALYSISSYSTEMS OF RECORD DATA SCIENTISTS DECISION MAKERS BUSINESS USERS DATA ANALYSTS
- 13. Copyright (C) 2017 451 Research LLC New systems of engagement require intelligence SYSTEMS OF ANALYSISSYSTEMS OF RECORD SYSTEMS OF ENGAGEMENT DATA ANALYSTS DATA SCIENTISTS DECISION MAKERS BUSINESS USERS SYSTEMS OF INTELLIGENCE
- 14. Copyright (C) 2017 451 Research LLC Batch processing as a barrier to intelligence SYSTEMS OF ANALYSISSYSTEMS OF RECORD SYSTEMS OF ENGAGEMENT DATA ANALYSTS DATA SCIENTISTS DECISION MAKERS BUSINESS USERS SYSTEMS OF INTELLIGENCE
- 15. 15 Problems with what’s going on here What’s slowing the conversion of data to insight? 1. Monolithic ETL systems 2. “Waterfall” changes 3. No shared responsibility 4. Singular unit of scale 5. Testing....???
- 16. The case for Microservices for Continuous Data Integration 1
- 17. Copyright (C) 2017 451 Research LLC The Case for Continuous Data Integration SYSTEMS OF ANALYSISSYSTEMS OF RECORD SYSTEMS OF ENGAGEMENT DATA ANALYSTS DATA SCIENTISTS DECISION MAKERS BUSINESS USERS SYSTEMS OF INTELLIGENCE
- 18. Copyright (C) 2017 451 Research LLC Continuous Data Integration The key concepts of a continuous integration and delivery process can be applied to the process of developing, deploying and managing data integration pipelines, resulting in pipelines that are responsive to changing business and data processing requirements.
- 19. for Data Integration Before Cloud ● Servers were expensive. ● Software was hard to acquire and slow to deploy. ● Nightly batches ● Integration in large rollouts and required specialized skills. Today ● Lower compute cost ● Software is core to business value, developed in-house. ● Higher data volumes ● Scale-out architectures ● Streaming data sources Why Microservices Integration flows can leverage this by operating as microservices that scale independently, and are reusable, and interoperate over message queues.
- 20. Spring Cloud Data Flow Spring Cloud Data Flow is a Microservices toolkit for building data integration and real- time data processing pipelines. Pipelines consist of Spring Boot apps, using Spring Cloud Stream for events or Spring Cloud Task for batch processes. The Data Flow server provides interfaces to compose and deploy pipelines onto a runtime like Kubernetes or Cloud Foundry.
- 21. Day 2: People and Platforms 2
- 22. Copyright (C) 2017 451 Research LLC What Exactly Needs to be Transformed Digitally? Process Transforma-on Informa-on Transforma-on Pla0orm Transforma-on ! More than a technical shi., but a cultural one ! Focus on collabora3on—employees but also customers, partners, suppliers ! Agile methods for so.ware development ! Gathering data and lots of it in various means and methods ! Mul3ple communica3on points on mul3ple devices ! Leveraging data with advanced analy3cs ! IT moving from cost center to so.ware enabler ! Organiza3ons needs systems of engagement—tools and systems for omnichannel interac3on ! Integra3on with legacy systems of record
- 23. 23 Methodologies Deployment Sparingly at designated times Ready for prod at any time Architecture Technologies Operations App Server on Machine Containers, Public / Private / Hybrid Cloud Monolithic Systems Microservices / Composite app Linear / Sequential Agile DevOps CI / CD Pipelines Many tools, ad hoc automation Manage services, not servers Software Engineering Practices are Evolving
- 24. So, Now What? 2
- 25. Align your Data engineering and development teams Identify data sources, processes, and destinations Focus on system of engagement in need of intelligence
- 26. 26 Thank You!