Integration Reloaded - Integration solutions based on reactive principles
1 like402 views
The document discusses the evolution of integration solutions, highlighting the necessity of adopting reactive principles in response to increasing complexity in technology and business demands. It describes various reactive patterns such as backpressure, circuit breakers, and bulkheads, which enhance agility and resilience in integration architectures. Furthermore, it emphasizes new integration approaches using technologies like Akka, Kafka, and serverless computing to meet modern business requirements effectively.
![Red Expert Alliance
https://www.redexpertalliance.com
New digital business models in bicycle rental
[Rainer Klute]](https://image.slidesharecdn.com/20181025integrationreloadedintegrationsolutionsbasedonreactiveprinciples-181027104807/85/Integration-Reloaded-Integration-solutions-based-on-reactive-principles-38-320.jpg)
![Red Expert Alliance
https://www.redexpertalliance.com
Vandalism is the Hulk of our business model
[Marcus Pink]](https://image.slidesharecdn.com/20181025integrationreloadedintegrationsolutionsbasedonreactiveprinciples-181027104807/85/Integration-Reloaded-Integration-solutions-based-on-reactive-principles-44-320.jpg)
Integration Reloaded - Integration solutions based on reactive principles
- 1. Red Expert Alliance https://www.redexpertalliance.com Integration Reloaded Integration Solutions Based on Reactive Principles
- 2. Red Expert Alliance https://www.redexpertalliance.com About: Arturo Viveros Arturo Viveros • Principal Middleware Architect at Sysco MW • Oracle ACE • Oracle Dev Champion • Twitter: @gugalnikov • Blogs: ‣http://blog.sysco.no/ ‣https://soamythbusters.wordpress.com ‣http://oracletechnocore.blogspot.no
- 3. Red Expert Alliance https://www.redexpertalliance.com About: Sven Bernhardt Sven Bernhardt • Senior Solution Architect at OPITZ CONSULTING Deutschland GmbH • Oracle ACE • Twitter: @sbernhardt • Blogs: ‣http://thecattlecrew.net ‣https://svenbernhardt.wordpress.com
- 5. Red Expert Alliance https://www.redexpertalliance.com Agenda Status Quo Integration01 02 Use case04 Architectural thoughts03 Summary05 Reactive Patterns
- 6. Red Expert Alliance https://www.redexpertalliance.com What is integration about? • Simplified: Get Data from System A and send it to System B System A System B XX• Message/Protocol Transformation
- 7. Red Expert Alliance https://www.redexpertalliance.com What is integration about? • Message Splitting • Routing System A System C System B
- 8. Red Expert Alliance https://www.redexpertalliance.com ESB solutions as basic architecture element global internal specific External Interfaces – Proxy Servics Integration Logic Internal Interfaces – Business Interfaces Database
- 9. Red Expert Alliance https://www.redexpertalliance.com Limited and expensive Scalability Loadbalancer ESB 1 ESB 2 ESB 3 Database
- 10. Red Expert Alliance https://www.redexpertalliance.com Caught in a technology Trap SOA Cloud Service Fixed Technologies Set <XML /> { „dialect“:“JSON“ }
- 11. Red Expert Alliance https://www.redexpertalliance.com Limited agility • Hard to implement Continuous Integration / Continuous Delivery • Complex infrastructure • Missing automated testing capabilities • Expert knowledge needed • Implementation • Deployment • Error analysis http://geek-and-poke.com/
- 12. Red Expert Alliance https://www.redexpertalliance.com But the world is getting more complex!
- 13. Red Expert Alliance https://www.redexpertalliance.com So we’re Constantly Dealing with Overstretched Integration Components / Patterns • ESB • JMS • Application Server • JCA Adapter • SOAP / XML • Etc. http://geek-and-poke.com/
- 14. Red Expert Alliance https://www.redexpertalliance.com Question How to deal with this new complexity efficiently? (By the way it doesn‘t mean that everything we did before was wrong!)
- 15. Red Expert Alliance https://www.redexpertalliance.com Agenda Status Quo Integration01 02 Use case04 Architectural thoughts03 Summary05 Reactive Patterns
- 16. Red Expert Alliance https://www.redexpertalliance.com Reactive Principles Elastic Responsive Message Driven Resilient Value Means Form https://www.reactivemanifesto.org/
- 17. Red Expert Alliance https://www.redexpertalliance.com Actor model • Advantages: • Useful for realizing reactive patterns • Fault tolerant • Distributed execution • Scalability Source SinkActor autonomous Async - Boundary
- 18. Red Expert Alliance https://www.redexpertalliance.com Adopting Reactive Patterns (i.e. Ivy Pattern)
- 19. Red Expert Alliance https://www.redexpertalliance.com Actor hierarchies Source SinkActor autonomous
- 20. Red Expert Alliance https://www.redexpertalliance.com Actor Balancing Topic Actor Actor Router Queue Queue Actor Actor MasterQueue Actor Actor
- 21. Red Expert Alliance https://www.redexpertalliance.com Reactive Patterns: Backpressure • Systems should be designed and implemented to react assertively under sustained load • This is often neglected although such conditions are usually expected and highly predictable • Negotiating back pressure usually requires some sort of Flow Control • Which in simpler terms is the ability to say “hey, slow down!!”, or even “stop it!!”
- 22. Red Expert Alliance https://www.redexpertalliance.com Non-Blocking Backpressure Concept Source SinkActor autonomous Give me 3 messages! Give me 20 messages!
- 23. Red Expert Alliance https://www.redexpertalliance.com Serveral ways to handle Backpressure • Dynamic push / pull • Static / Dynamic Quotas • Buffers • Ack / Nack • Throttling • Caching • Circuit breaker • Scaling
- 24. Red Expert Alliance https://www.redexpertalliance.com Reactive Patterns: Circuit Breaker https://blog.risingstack.com/designing-microservices-architecture-for-failure/ • Used in the industry to protect electronic components • In software development • Protect resources • Help resources to auto-recover from failures • Prevent hanging operations
- 25. Red Expert Alliance https://www.redexpertalliance.com Reactive Patterns: Bulkhead • Used in the industry to partition a ship into sections • In software development • Segragate resources • Protect limited resources from being exhausted
- 26. Red Expert Alliance https://www.redexpertalliance.com Avoiding single points of failure is key for “reactiveness”
- 27. Red Expert Alliance https://www.redexpertalliance.com Important Concept: a chain is no stronger than its weakest link
- 28. Red Expert Alliance https://www.redexpertalliance.com Agenda Status Quo Integration01 02 Use case04 Architectural thoughts03 Summary05 Reactive Patterns
- 29. Red Expert Alliance https://www.redexpertalliance.com Implementing Reactive Patterns with Akka • Akka Streams • Ideal for Fast Data • Seamless Integration • Abstraction to configure end-to-end Backpressure • Fully Asynchronous & non-blocking • Bulkheads & Circuit Breaker patterns are also supported
- 30. Red Expert Alliance https://www.redexpertalliance.com Implementing Reactive Pattern with Kafka • Distributed streaming platform • Dynamic durable buffer • Horizontally scalable • Load distribution • No data loss settings (Producer, Consumer, Broker) • Quotas
- 31. Red Expert Alliance https://www.redexpertalliance.com Next-gen Integration Platform Sysco reactive Integration platform
- 32. Red Expert Alliance https://www.redexpertalliance.com Demo: Backpressure • Fast Producer, Fast Consumers • Fast Producer, Fast and Slow Consumers • Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.dropNew • Fast Producer, Fast and Slow Consumers with buffering: OverflowStrategy.backpressure
- 33. Red Expert Alliance https://www.redexpertalliance.com Implement Integration with Serverless/FaaS • Function-as-a-Service (FaaS) • Atomar step within a certain business capability • Example: Bicycle rental - Process incoming Damage events • Serverless • No need to care about servers • Scale-out functions automatically on-demand • You only pay processing time
- 34. Red Expert Alliance https://www.redexpertalliance.com Implement Integration with Serverless/FaaS System A System B X f(x) getData f(x) sendDataf(x) transformData
- 35. Red Expert Alliance https://www.redexpertalliance.com When is FaaS adequate for integration? • Typical use cases: • Batch processing, which takes place on regular schedules • Event processing • Considerations to make: • Number of invocations • Keep runtime short • Serverless Platforms will terminate long-running Functions • Don‘t block HW resources for too long
- 36. Red Expert Alliance https://www.redexpertalliance.com Integration with Fn Project • Implement functions • F(x) getData • F(x) transformData • F(x) sendData • Orchestrate functions using Fn Flow • Alternatively: Use Topics in between the functions
- 37. Red Expert Alliance https://www.redexpertalliance.com Agenda Status Quo Integration01 02 Use case04 Architectural thoughts03 Summary05 Reactive Patterns
- 38. Red Expert Alliance https://www.redexpertalliance.com New digital business models in bicycle rental [Rainer Klute]
- 39. Red Expert Alliance https://www.redexpertalliance.com Happy Path: Renting a bike Scan a QR Codes at bike rental station with an mobile app Book the rental bike Remove the bicycle from the station and drive off Open lock at rental station
- 40. Red Expert Alliance https://www.redexpertalliance.com Use Case: Bonus Action „After renting a bike for a fee for the 10th time you get a 20% discount and can get a free coffee at the next Mermaid Coffee during your tour at our expense.“
- 41. Red Expert Alliance https://www.redexpertalliance.com What does this mean for the application? • Connection to CRM and ERP necessary • Calculation of different cost rates on the basis of historical data • Connection of all branches of the external partner (Mermaid Coffee) • Near real-time - cyclists must still have rented a bike to redeem their coffee • Payment flows between partners, so that a high degree of robustness is necessary
- 42. Red Expert Alliance https://www.redexpertalliance.com Solution approach: Actors & Reactive Principles Enable bonuses? Buffer Topic Get data from ERP Get data from CRM Inform Partner Inform Custome r Topic
- 43. Red Expert Alliance https://www.redexpertalliance.com Use Case: Damage 3 out of 5 bikes at the same rental station fell over in 2 minutes???
- 44. Red Expert Alliance https://www.redexpertalliance.com Vandalism is the Hulk of our business model [Marcus Pink]
- 45. Red Expert Alliance https://www.redexpertalliance.com Solution approach: Fast Data & Integration Analyze Events Buffer Topic Inform technician Damage events New Vandalism event Process Damage claims Topic
- 46. Red Expert Alliance https://www.redexpertalliance.com Agenda Status Quo Integration01 02 Use case04 Architectural thoughts03 Summary05 Reactive Patterns
- 47. Red Expert Alliance https://www.redexpertalliance.com Summary • Integration is still a relevant discipline nowadys, but the way how to implement it has chnged • Existing integration approaches based on ESB solutions are still valid, but • Businesses getting disrupted, are volatile and techologies is emerging rapidly, so the architecture needs to consider this • Architectures must address this accordingly