When to use What? A look at choosing Integration Technology

Editor's Notes

• #3: Who am i
• #4: Where are we / problem
• #5: Previous book (cover) Technology count: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB Relay, StreamInsight Technology count TODAY: WCF, WF, AppFabric, BizTalk, ESB, SSIS, SSSB, Sync Framework, Azure SB relay, Azure SB Brokered, Azure VMs, Azure Notification Hubs, BizTalk Services, StreamInsight LET’S TALK THROUGH SOME TRENDS THAT EMERGED SINCE THIS BOOK
• #7: Source: http://www.flickr.com/photos/davenielsen/5718247676/sizes/l/ Cloud computing Change to the traditional computing paradigm Host matters less; elastic, on-demand Accepted part of an organization’s app portfolio New types of applications and scenarios Widely distributed Mix of hosting providers Different integration scenarios Cloud to cloud, ground to cloud, cloud to ground
• #8: Source: http://www.flickr.com/photos/mattandbeccasteudelwedding/9067696311/sizes/l/ Micro Services Something championed by ThoughtWorks a while back, and now something Netflix (and others) do a lot of Mini applications/services No heavy, complex services Very specific services that are easy to consume and weave together Think of cloud providers who have distinct services for provisioning services, autoscale, etc
• #9: Source: http://www.flickr.com/photos/24752578@N02/5589897831/sizes/l/ Lightweight Integration It’s not just about a powerful, centralized integration server That’s of course always been the case, but new attention paid to simplified messaging Something between a basic queue and a full blown ESB New tools, new way to connect lightweight services together
• #10: Source: http://www.flickr.com/photos/anekoho/9638930676/sizes/h/ NoSQL endpoints New types of endpoints, often outside your direct control Relational databases are less important now than ever Still a place for them, but major companies are finding they don’t need them We’ve replaced our SQL backend with one To the integrator, this adds new scenarios
• #11: Source: http://www.flickr.com/photos/peternijenhuis/6959653638/sizes/h/ Automation DevOps and the cultural movement to unprecedented synergy between Dev and Ops teams Automated builds, continuous integration, continuous deployment Possible to get rid of snowflake servers, and move to immutable servers
• #13: Source: http://www.flickr.com/photos/suckamc/5907306188/sizes/l/ We want to be at a place where we have the simplest toolset possible, but one that meets our needs This continued expansion of available integration services is logical, when considering these trends There are more and more tools for specific scenarios (micro-integration services), and fewer monolithic tools that serve EVERY purpose
• #14: How do we get there? Walk through a decision framework Talk through the available technologies Apply these principles
• #15: What’s a decision framework? Way to analyze a situation and identify the right fit There cannot be a massive decision matrix that drops the correct solution in your lap Instead, there are multiple dimensions to consider, and those may be unique to each organization and situation Steps First, let’s discuss the sources that describe our situation Then, we’ll talk about the 4 dimensions to discuss
• #16: Input into a decision … Functional requirements What is the application supposed to do? Business requirements
• #17: Input into a decision … Non functional requirements Describe what the system itself needs to do to support the functional requirements security, availability, maintainability, compliance, exception handling, etc You can argue if these are actually functional requirements, and in some cases they are
• #18: Input into a decision … Derived requirements Ones that aren’t explicit stated, but can be derived from what’s needed Based on a functional requirement to provide a certain type of data, I might derive a requirement to persist data in a warehouse for long range analytics
• #19: Input into a decision … Organization direction/strategy This is the least transferrable! What are the company goals, investments? These are often fixed, and non-negotiable NOW, LET’S TALK ABOUT THE DECISION CRITERIA ITSELF
• #20: Solution design – design patterns required Data volumes, data set size, guaranteed delivery, data sources/destinations, latency, sync/async, data processing (workflow, quality operations), SSO, failure handling MODERN UPDATES … (reliability of endpoint, location of endpoint, constraints imposed by source/destination, circuit breakers)
• #21: Solution development – what it takes to build the solution Developer availability, learning curve, developer setup, IDE and tool integration, support system, source control integration, automated builds MODERN UPDATES … (works in cloud, web-based IDEs, continuous integration/delivery support)
• #22: Solution operations – what it takes to maintain the solution Event logging, tuning ability, handle bursts and failures, backup/restore routines, support tools, versioning capabilities, scriptability MODERN UPDATES … (immutable runtimes, rapid releases/rollbacks, works in distributed fashion)
• #23: Organizational considerations – enterprise standards and org direction Long term solution, vendor support, uses existing software investment, limited budget impact, build vs buy compatibility, risk tolerance, speed to market MODERN UPDATES … (OSS, commodity skill sets, easy/no upgrade)
• #25: Product: BizTalk Server Tag Line: Comprehensive application integration Most Recent Release: 4/13 Capabilities: Durable messaging, orchestration, technology adapters, trading partner management, BAM, BRE, etc Core Use Cases: Async messaging-based integration between applications; stateful, long running workflow processes that coordinate interactions between systems Buy-Hold-Sell: Buy
• #26: Product: ESB Toolkit Tag Line: “Dynamic messaging” Most Recent Release: 4/13 with BizTalk 2013 Capabilities: Dynamic routing, services for mapping and pipelines Core Use Cases: Diverse message paths; message flow modeling, supremely loose coupling, centralized exception management Buy-Hold-Sell: Soft Buy
• #27: Product: Windows Server AppFabric Tag Line: Advanced service hosting Most Recent Release: 3/12 Capabilities: Caching, advanced WCF and WF hosting; troubleshooting; persistent workflows Core Use Cases: WCF service management, hosting long running WF services Buy-Hold-Sell: Hold
• #28: Product: Service Bus for Windows Tag Line: On-premises durable messaging Most Recent Release: 10/13 Capabilities: Running messaging aspects of Azure Service Bus on-premises; topics and queues; manage with new Windows Azure Pack on Windows Core Use Cases: Start work on premises and move to cloud; basic pub-sub messaging; Buy-Hold-Sell: Hold
• #29: Product: Workflow Manager Tag Line: On-premises workflow host Most Recent Release: 3/13 (CU) Capabilities: multi-tenancy, tracking and monitoring, instance management, versioning support Core Use Cases: Host and manage WF apps; Buy-Hold-Sell: Hold http://msdn.microsoft.com/library/jj193528
• #30: Product: WCF Tag Line: Sophisticated service framework Most Recent Release: Capabilities: Contract first service design for REST and SOAP services; configuration-centric service model that is extreme (overly?) extensible Core Use Cases: Building SOAP based web services; Sync or async Buy-Hold-Sell: Soft Buy
• #31: Product: ASP.NET Web API Tag Line: Modern web services Most Recent Release: 1/14 Capabilities: REST/HTTP service design; extensible model; Odata support Core Use Cases: web service design Buy-Hold-Sell: Buy
• #32: Product: Windows Workflow Tag Line: Lightweight, durable processes Most Recent Release: Capabilities: declarative models that describe a process; built-in and custom activities; durable workflow; service endpoints Core Use Cases: long running workflows; exposing business processes as services Buy-Hold-Sell: Soft Buy
• #33: Product: SSIS (also Master Data Services?) Tag Line: Bulk data transfer Most Recent Release: SQL 2014 (no big changes) Capabilities: Data integration and data transformation; lots of data sources (files, FTP, HTTP, SMTP, RDBMS); bulk movement Core Use Cases: Buy-Hold-Sell: Buy
• #34: Product: StreamInsight Tag Line: Event stream processing Most Recent Release: 6/12 Capabilities: high volume pattern matching against event streams; adapter model; Iobservable, Ienumerable support; multiple hosting options; LINQ queries Core Use Cases: situational awareness from data streams Buy-Hold-Sell: Soft Hold
• #35: Product: Windows Azure Virtual Machines – BizTalk Image Tag Line: Full BizTalk in the Cloud Most Recent Release: June 2013 Capabilities: Run full BizTalk Server in the cloud; no HA with SQL clustering Core Use Cases: Dev/test scenarios Buy-Hold-Sell: Hold
• #36: Product: Windows Azure Service Bus Relays Tag Line: Services that span boundaries Most Recent Release: Capabilities: Traverse networks and securely expose sync and async WCF services Core Use Cases: partner integration, mobile device integration, remote worker tools, cloud to cloud integration Buy-Hold-Sell: Strong Buy
• #37: Product: Service Bus Brokered Messaging Tag Line: Durable messaging in the cloud Most Recent Release: Capabilities: Durable messaging through queues (single reader), and topics (broadcast with subscriptions) Core Use Cases: partner integration, mobile integration, cloud to cloud integration Buy-Hold-Sell: Buy
• #38: Product: Service Bus Notification Hubs Tag Line: Scalable message distribution Most Recent Release: Capabilities: Broadcast messages to mobile devices and Windows 8; Core Use Cases: Buy-Hold-Sell: Buy
• #39: Product: Windows Azure BizTalk Services Tag Line: Lightweight cloud message broker Most Recent Release: Capabilities: Link applications via cloud-hosted broker; source/destination adapters; strong data mapper Core Use Cases: connecting partners, cloud apps Buy-Hold-Sell: Strong Hold
• #40: MANY other products that work well in a Windows and cloud world Brokers like Apache Kafka Queue technologies like RabbitMQ, or Amazon SQS Cloud-based integration tools like SnapLogic and Informatica Cloud SaaS-integrated services for message publication and workflow E.g. Dynamics CRM Online or Salesforce.com
• #43: Options … Use messaging and use BizTalk to send messages to vendor endpoints PRO: Good tracking, chance for enrichment before sending out; guaranteed delivery CON: hard to quickly change endpoints; puts responsibility on internal team to manage/scale; Use messaging and use BizTalk to send to Topics, local DB, and notification Hubs Use messaging and send loans directly to SB topics (and internal DB) from WF PRO CON Send loans to SB topic and one listener is internal DB PRO: Simple, offload responsibility for delivery CON: No coordination; chance of race condition; no notification for vendors of new messages Send loans to SB topic, one listener is internal DB, and also use Notification Hubs for alerting Provide REST-ful query API (exposed via SB) on loan DB and use notification hubs to send messages to providers to query and provide their best offer PRO: Simple; no replication; easy to onboard new vendors; offload availability to 3rd party; durable; CON: Not event based; need to chain together way to notify vendors; no orchestration or tracking; limited visibility into vendor processing
• #44: Let’s consider the option that leverages the Service Bus for all this Send a message to Notification Hubs so very easy to broadcast info and push alerts Use Topics for scalable, durable messaging Subscribe ourselves to update internal systems