Integration architectures based on Microservices, APIs and events
0 likes71 views
The document discusses the evolution and challenges of integration architectures based on microservices, APIs, and events, emphasizing the need for modernization due to the obsolescence of legacy systems. It outlines strategic options for integrating new technologies while ensuring data quality, governance, and operational excellence. A shift from monolithic to cloud-native architectures is advocated, highlighting the importance of a platform-centric approach to integration in order to support digital transformation.
Integration architectures based on Microservices, APIs and events
- 1. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 1 Gummersbach, June 3, 2023 Sven Bernhardt INTEGRATION ARCHITECTURES BASED ON ΜICROSERVICES, APIS & EVENTS
- 2. © OPITZ CONSULTING 2023 / Öffentlich THAT‘S ME Integration Architectures based on µServices, APIs & Events 2 Sven Bernhardt Cloud-Native enthusiast, API & integration geek. Always curious how new technologies and concepts can help to make things more valuable and efficient. @sbernhardt https://svenbernhardt.wordpress.com/ Chief Architect / Integration Evangelist OPITZ CONSULTING Deutschland GmbH
- 3. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 3 WHY INTEGRATION? 01 INTEGRATION: YESTERDAY AND TOMORROW 02 OC INTEGRATION STORY 03 CONCLUSION 04
- 4. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 4 WHY INTEGRATION? 01
- 5. © OPITZ CONSULTING 2023 / Öffentlich Seite 5 This is what it looks like sometimes/often/always in application landscapes... https://www.youtube.com/watch?v=iXaw70X7wb4&t=2s Integration Architectures based on µServices, APIs & Events
- 6. © OPITZ CONSULTING 2023 / Öffentlich „STRUKTURELLE ZUKUNFTSUNFÄHIGKEIT“ 6 ¢ End of Support ¢ Lack of maintainability ¢ Declining benefits ¢ Dying knowledge ¢ Lack of agility / Implementation backlog ¢ Declining data quality ¢ Rising costs ¢ … „Despite its benefits for companies, old software is structurally unsustainable. This is due to their characteristics and the resulting risks for the actual business. “ (Lünendonk 2015) Integration Architectures based on µServices, APIs & Events
- 7. © OPITZ CONSULTING 2023 / Öffentlich 20% Expiring sourcing contracts with IT service providers. 71% Pressure to innovate in IT in order to better support the business. 88% Integration capability of new digital solutions into legacy software. 92% New technologies (e.g. AI, digital marketing, etc.) are not supported by legacy. DRIVERS OF MODERNIZATION. Integration Architectures based on µServices, APIs & Events 7 Fit for digital transformation Status quo and goals for legacy modernization and cloud migration (Lünendonk Studie 2019) 31% Maintenance, care and further development is no longer guaranteed by a lack of skills. DIGITAL PRODUCT
- 8. © OPITZ CONSULTING 2023 / Öffentlich STRATEGIC OPTIONS FOR MODERNIZATION Integration Architectures based on µServices, APIs & Events 8 Current situation ? Continuation Monolith today tomorrow day after tomorrow Retention Platformshift / Generator Concepts ? Monolith Crushing Reengineering Custom off-the-Shelf Relief Change Deprecated applications (mostly monolithic): • Legacy Java (pre-cloud-native) • Legacy App Server • Cobol • RPG • Oracle Forms • Database monoliths Condition: Structurally unsustainable Save time, postpone problem to later Modernized applications (mostly cloud-native): • Containerized • Aligned with Business Capabilities • Maintenance-friendly • Design for Change Condition: Structurally sustainable Is there new, cost-optimized software on the market?? The "right" way can only be decided individually. e.g. Lift&Shift, CI/CD, dyn. Runtime env
- 9. © OPITZ CONSULTING 2023 / Öffentlich 10 PRODUCT-CENTRIC APPLICATION WORLD DIGITAL PRODUCT Integration Architectures based on µServices, APIs & Events Digital Product Monolith Digital Product Business Domain Shared Application Platform Shared Services (Foundation) incl. integration platform Hybrid, flexible infrastructure incl. Cloud Services Standard (On-Prem) Business Domain Business Domain Digital Product Standard (On-Prem) SaaS-Cloud Standard SaaS integration is the foundation for it-modernization -> Necessary to clean-up! Gartner: Composable Enterprise
- 10. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 11 INTEGRATION: YESTERDAY AND TOMORROW 02
- 11. © OPITZ CONSULTING 2023 / Öffentlich INTEGRATION? IT’S JUST ABOUT BOXES AND LINES! Goals: ¢ Enabling systems to talk to each other ¢ Ensure that data is where it is needed when it is needed in the form in which it is needed Gregor Hohpe A small line. That‘s where the beef is! Integration Architectures based on µServices, APIs & Events 12
- 12. © OPITZ CONSULTING 2023 / Öffentlich INTEGRATION AREAS User-centric apps („users”) Real-life objects („things”) Cloud On premise Cloud to user On premise to user Cloud to thing On premise to thing Cloud to cloud On premise to on premise ¢ There are seven typical integration areas in IT landscapes Cloud to on premise 1 2 3 4 5 6 7 Integration Architectures based on µServices, APIs & Events 13
- 13. © OPITZ CONSULTING 2023 / Öffentlich INTEGRATION CHALLENGES ARE MULTIDIMENSIONAL Integration Architectures based on µServices, APIs & Events 14
- 14. © OPITZ CONSULTING 2023 / Öffentlich BASED ON PATTERNS THAT ARE STABLE SINCE DECADES Content-based Router Message Filter Splitter Aggregator Normalizer Resequencer https://www.enterpriseintegrationpatterns.com Integration Architectures based on µServices, APIs & Events 15
- 15. © OPITZ CONSULTING 2023 / Öffentlich USUAL STATE OF INTEGRATION ARCHITECTURE – IF THERE‘S ANY… ¢ Monolithic architecture ¢ Often classic JEE deployments ¢ Integrations share single execution process ¢ Limited freedom of technology choice ¢ Core component: ¢ ESB Platform (Oracle Service Bus, Sonic ESB) ¢ Reverse Proxy for handling external traffic ¢ Focus on horizontal integration challenges Integration Architectures based on µServices, APIs & Events 16
- 16. © OPITZ CONSULTING 2023 / Öffentlich FROM CENTRALIZED TO DECENTRALIZED ARCHITECTURES Integration Architectures based on µServices, APIs & Events 17 Centralized STATIC ON-PREM MONOLITH VIRTUAL MACHINES MANUAL CHANGE PROCESS Decentralized DYNAMIC CLOUD / MULTI-CLOUD MICROSERVICES / SERVERLESS CONTAINERS, KUBERNETES AUTOMATED CI/CD TOOL CHAIN # Services & APIs CONTROL AND VISIBILITY
- 17. © OPITZ CONSULTING 2023 / Öffentlich PLATFORMS NEED TO REFLECT THIS CHANGE 18 Integration Architectures based on µServices, APIs & Events CLASSIC API GATEWAY MODERN API GATEWAY MONOLITHIC APP API API MANAGEMENT CLIENT PRIVATE TRAFFIC PUBLIC TRAFFIC GW DP CLIENT PUBLIC TRAFFIC GW DP API APP API APP API APP API APP API APP GW DP GW DP ✓ High Performance ✓ Multi-cloud and Hybrid ✓ Kubernetes-native ✓ CI/CD automation ✓ Modern protocols
- 18. © OPITZ CONSULTING 2023 / Öffentlich Adapter Processes Systems APIs INTEGRATION NEEDS TO CONSIDER SYSTEMS, DATA & PROCESSES 19 Integration Architectures based on µServices, APIs & Events Vertical Integration Horizontal Integration Data
- 19. © OPITZ CONSULTING 2023 / Öffentlich CLOUD-NATIVE INTEGRATION – THIS IS THE WAY… ¢ Best-of-breed approach ¢ Cloud-native architecture ¢ Containerization ¢ Kubernetes as Core runtime platform ¢ Single execution process per integration ¢ Core components: ¢ Distributed, lightweight ESB ¢ Event Hub for async integration ¢ API Gateway for sync integration ¢ Focus on horizontal as well as vertical integration challenges Integration Architectures based on µServices, APIs & Events 20
- 20. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 21 TECHNOLOGY BASE - USE OPEN INDUSTRY STANDARDS https://landscape.cncf.io/
- 21. © OPITZ CONSULTING 2023 / Öffentlich 22 OC INTEGRATION STORY 03 Integration Architectures based on µServices, APIs & Events
- 22. © OPITZ CONSULTING 2023 / Öffentlich CRUSHING THE MONOLITH Seite 23 Integration Architectures based on µServices, APIs & Events Was completely transferred to SCS Own microservices/SCSs with web UI, mobile UI and chatbot interface (teams and mobile) with API connection Introduction of standard software (Salesforce, etc.)
- 23. © OPITZ CONSULTING 2023 / Öffentlich AS-IS SITUATION – “SPAGHETTI” INTEGRATION Integration Architectures based on µServices, APIs & Events 24 SharePoint PVS Doxis FiBu LoBu Active Directory DWH assyst OpenScape HR4You Oscar Lohn Polen Onesto Exchange FileServer Steak TLMT Website + Tracking
- 24. © OPITZ CONSULTING 2023 / Öffentlich DESIRED SITUATION – STRUCTURED INTEGRATION Integration Architectures based on µServices, APIs & Events 25 Doxis Active Directory DWH assyst OpenScape Oscar Lohn Polen Onesto Exchange FileServer Steak Website + Tracking CRM OC Integration Platform (OCIP) ERP SharePoint TLMT 3rd Party Cloud(s) HR Business Partners OC employees
- 25. © OPITZ CONSULTING 2023 / Öffentlich GOALS OC INTEGRATION PLATFORM Integration Architectures based on µServices, APIs & Events 26 ¢ Ensure data integrity and quality during PVS replacement ¢ Simplification of the OC system landscape ¢ Traceability of OC integration paths ¢ Improved governance and compliance management (APIs, data structures, etc.) ¢ Creation of a technological basis for ensuring sufficient data quality in the future ¢ Increase operational excellence ¢ Ensuring an evolutionary architecture ¢ Future-proof ¢ Extensibility ¢ Robustness and Resilience ¢ Scalability and elasticity
- 26. © OPITZ CONSULTING 2023 / Öffentlich VISION OC INTEGRATION PLATFORM Integration Architectures based on µServices, APIs & Events 27 ¢ Ensuring an evolutionary architecture (Design4Change) ¢ Future-proof ¢ Extensibility ¢ Robustness and Resilience ¢ Scalability and elasticity ¢ OCIP best-of-breed with characteristics ¢ Modern ¢ Automated ¢ Integrated ¢ Secure “Als Rückgrat unserer #zukunftswirksamen IT- Landschaft setzen wir auf eine dynamikrobuste Integrationsplattform, um unsere Systeme, Daten und Prozesse zu integrieren: OC|SPINE”
- 27. © OPITZ CONSULTING 2023 / Öffentlich OC SPINE IS BUILT ON CLOUD-NATIVE PRINCIPLES … 28 Integration Architectures based on µServices, APIs & Events
- 28. © OPITZ CONSULTING 2023 / Öffentlich … AND TECHNOLOGIES Integration Architectures based on µServices, APIs & Events 29 ¢ Open-Source tools from CNCF landscape ¢ Key technology components ¢ API Gateway (Kong) ¢ Event Hub (Confluent Cloud) ¢ Schema Registry (Confluent Schema Registry) ¢ Apicurio (API Registry) ¢ Secrets Management (HashiCorp Vault) ¢ Identity Management (Keycloak) ¢ Service Mesh (Kuma) ¢ Application-level monitoring is done with Grafana Stack (Grafana Cloud)
- 29. © OPITZ CONSULTING 2023 / Öffentlich KEY CONCEPTS Integration Architectures based on µServices, APIs & Events 30 ¢ Loosely-coupled service architecture ¢ System-independent domain model ¢ Transparent error handling ¢ Centralized observability
- 30. © OPITZ CONSULTING 2023 / Öffentlich SERVICE CATEGORIES 31 ¢ Presentation Services (Single-purpose APIs) ¢ Expose tailored information set for specific clients ¢ Use the information provided by Domain services ¢ Business Services (Multi-purpose APIs) ¢ Implement business resp. domain logic ¢ Operate on a domain object model ¢ Combine information provided by Adapters ¢ Connectivity Services (Adapter) ¢ Implement system-specific logic ¢ Convert data & Protocols ¢ Expose a domain model Integration Architectures based on µServices, APIs & Events
- 31. © OPITZ CONSULTING 2023 / Öffentlich EXAMPLE: SALES – PVS BACKWARD INTEGRATION 32 Integration Architectures based on µServices, APIs & Events
- 32. © OPITZ CONSULTING 2023 / Öffentlich “LEGO”-LIKE COMPONENT ARCHITECTURE 33 Integration Architectures based on µServices, APIs & Events EventHub PVS DWH Other Systems API Gateway ChangedDataEvent DomainEvent DomainEvent DomainEvent InsertChangedData InsertChangedData GetSalesforceObject GetDomainObject
- 33. © OPITZ CONSULTING 2023 / Öffentlich WITHOUT A SYSTEM-INDEPENDENT DOMAIN MODEL Integration Architectures based on µServices, APIs & Events 34 Präsentation 34 Account • CId : String • Name : String • … ErpKunde • K_ID : Number • C_ID : String • K_NAME : String • … DwhKunde • ID : Number • C_ID : String • FIRMEN_NAME : String • … Integration platform CRM ERP DWH Account -> DwhKunde Account -> ErpKunde
- 34. © OPITZ CONSULTING 2023 / Öffentlich CHALLENGES TO SOLVE Integration Architectures based on µServices, APIs & Events 35 ¢ Interdependencies between the systems ¢ Changes in data structures ¢ System-specific IDs for X-referencing ¢ X-referencing challenges may be addressed by introducing X-Ref functionality ¢ Domain knowledge in Product teams
- 35. © OPITZ CONSULTING 2023 / Öffentlich SOLUTION APPROACH Integration Architectures based on µServices, APIs & Events 36 Account • CId : String • UUID : String • Name : String • … ErpKunde • K_ID : Number • UUID : String • K_NAME : String • … DwhKunde • ID : Number • UUID : String • FIRMEN_NAME : String • … Integration platform CRM ERP DWH DomainAccount -> DwhKunde DomainAccount -> ErpKunde DomainAccount • UUID : String • Name : String • … Account -> DomainAccount Schema Management
- 36. © OPITZ CONSULTING 2023 / Öffentlich CENTRALIZED OBSERVABILITY Integration Architectures based on µServices, APIs & Events 37 ¢ Challenge: Distributed integration architecture that consists of multiple, independent services ¢ Provide a centralized 360-degree view of distributed integration architecture ¢ Infrastructure view / Platform health ¢ Application view / Integrations apps ¢ More efficient troubleshooting ¢ Key components: ¢ Grafana Stack, Datadog, EFK ¢ Service Mesh (Kuma, Istio, Consul Mesh)
- 37. © OPITZ CONSULTING 2023 / Öffentlich TRACING EXAMPLE VIEW (WITH APP INSTRUMENTATION) Integration Architectures based on µServices, APIs & Events 38
- 38. © OPITZ CONSULTING 2023 / Öffentlich EXAMPLE ERROR DASHBOARD (BASED ON APPLICATION LOGS) Integration Architectures based on µServices, APIs & Events 39
- 39. © OPITZ CONSULTING 2023 / Öffentlich TRANSPARENT ERROR HANDLING Integration Architectures based on µServices, APIs & Events 40 ¢ Different exception types ¢ Retryable (Network errors, Rate limits, etc.) ¢ Non-Retryable (Data errors, Business errors, etc.) ¢ Use of resilience patterns for auto-retry exceptions ¢ Circuit breakers ¢ Timeouts ¢ Retries ¢ Central view on errors ¢ Proactive notifications to platform and product teams
- 40. © OPITZ CONSULTING 2023 / Öffentlich 41 CONCLUSION 03 Integration Architectures based on µServices, APIs & Events
- 41. © OPITZ CONSULTING 2023 / Öffentlich LEARNING: THINK IN PLATFORMS! Integration Architectures based on µServices, APIs & Events 42 ¢ Integration is a shared responsibility ¢ Establish a platform team, responsible for: ¢ Maintain platform ¢ Operate platform ¢ Provide onboarding support (besides other services) ¢ Ensure Developer Experience ¢ Decentralized integration implementation ¢ Responsibility: Product teams ¢ Separation of concerns
- 42. © OPITZ CONSULTING 2023 / Öffentlich CONCLUSION Integration Architectures based on µServices, APIs & Events 43 ¢ Integration is the foundation for IT Modernization ¢ Structural inability threatens! ¢ Need for meaningful data, system, and process integration ¢ We need "Design4Change“ ¢ Think platform! Think product! ¢ Not boring stuff… Highly exciting from a technology perspective! ¢ A high degree of distribution. Everything is in motion. Many new fields of learning. ¢ Building solutions for an increasingly complex world needs professionalization A small line. That‘s where the beef is! DIGITAL PRODUCT
- 43. © OPITZ CONSULTING 2023 / Öffentlich Integration Architectures based on µServices, APIs & Events 44 Q & A
- 44. © OPITZ CONSULTING 2023 / Öffentlich IN CASE OF ANY QUESTIONS… Integration Architectures based on µServices, APIs & Events 45 Sven Bernhardt Chief Architect / Integration Evangelist | Oracle ACE Director OPITZ CONSULTING Deutschland GmbH Kirchstrasse 6, 51647 Gummersbach, Germany Phone: +49 172 2193529 Mail: sven.bernhardt@opitz-consulting.com Twitter: @sbernhardt Blog: https://svenbernhardt.wordpress.com