Integrate 2019 - Creating a Processing Pipeline with Azure Functions and AIS
- 1. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Wagner Silveira Microsoft MVP -Azure Creating a Processing Pipeline with AzureFunction and AIS
- 2. About me • Principal Integration Architect • Microsoft Azure MVP • MCSE Cloud Platform wsilveiranz wagner.silveira@notetoself.tech https://notetoself.tech
- 3. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Agenda • Case Study • Initial Solution • Updated Solution • Key Components • A year later – what changed? • Lessons Learned
- 4. Case Study
- 5. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London The solution requirements • Onboarding of external EDI messages • Should allow for different transports • Message need to be Validated, translated, enriched, routed • Control on retries and notification over failed messages • Auditing of initial message • Visibility of where a single message was in the process
- 8. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Reality Check • Big Payloads • EDIFACT schema not available • Operational costs • End to end Monitoring
- 10. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London How was this different from the initial solution? • Azure Functions • EDIFACT support via .NET Package • Claim check pattern • Dedicated instances • Azure Storage • Payload storage (claim check pattern) • Application Insights • End to End with custom events • Single technology for monitoring and notification
- 11. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Key Components
- 12. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Key Components • Claim check pattern • App Insights Custom Events
- 13. The claim check pattern • Publishing function separates content from metadata then: • Store content on a safe repository • Publish metadata + storage pointer + Reprocessing count to message engine • Subscribing processes receive metadata and retrieve content when required Content+ Metadata Service Bus TopicPublishing Function Storage Account Subscription Function Subscription FunctionMetadata Content Metadata Content
- 14. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London App Insights custom events • End to end traceability using a Correlation ID • Allow for exceptions to be captured the same way • Leverage Functions support for App Insights • Implemented as a reusable component App Insights Receive & Validate Transform Store & Enrich Route 1001 Received 1001 Validated 1001 Tranformed 1001 Stored 1001 Enriched 1001 Routed 1002 Received 1002 Failed Validation
- 15. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Hands on Time?
- 16. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Key Management • Leveraged MSI support where available • SQL Server • Storage • Key vault • Kept keys in key vault where MSI where not supported directly • Service Bus
- 17. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Exception Management & Retries on Functions • Catch blocks using a notify and throw pattern • Leverage Function SB binding peek-lock pattern • Adjusted Maximum Delivery Count on SB • Messages that failed where delivered to DLQ
- 18. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London • Logic Apps polling subscriptions DLQ every 6 hours • Each subscription DLQ could have it’s own logic • Reprocessing count • Resubmission logic • Notification logic • Email notification • Error blob storage Pipeline Topic Validate DLQ Persist DLQ Reprocessing < 3 Error Storage DLQ Management
- 19. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London One year later…
- 20. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London New technology candidates • Integration Services Engine • Azure Durable Functions • Event Grid
- 21. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London What about new features • Azure Functions Premium • Improved support for Key Vault • Improved support for MSI • Virtual Network support + Service Endpoints
- 22. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Lessons learned
- 23. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Review the fine print
- 24. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Operational cost is a design consideration
- 25. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Make the best of each technology
- 26. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London Think about the big picture
- 27. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London In Summary • Weight your technology options • Think about limitations, operational costs • Understand how your solution fit in the big picture • Understand the components roadmap
- 28. INTEGRATE 2019JUNE 3 - 5, 2019 etc.venues, London