IBM MQ cloud architecture blueprint
- 1. IBM MQ Cloud architecture blueprint David Ware Matt Roberts Chief Architect, IBM MQ Lead Architect, IBM MQ on Cloud May 2019
- 2. Overview This deck introduces recommended patterns and guidance for IBM MQ deployments in multi-location or multi-provider cloud scenarios Contents • Deployment options for IBM MQ in clouds • Architectural best practices
- 3. 3© 2019 IBM Corporation Example customer topology “before” Account #2Account #1 IBM Cloud region Account #3 Central IT / MQ team responsibility On-premises Key: Application Queue manger
- 4. Deployment options for IBM MQ in clouds
- 5. 5© 2019 IBM Corporation Hybrid cloud deployments On-premises Cloud#1 • Across customers and industries we see that both new and existing applications are moving to the cloud • Most enterprises are moving to a hybrid cloud topology with combinations of multiple cloud providers, locations and data centres • This opens up both new possibilities and new problems, but applications require messaging services more now than ever before • IBM MQ provides advanced capabilities not found in other offerings that enable you to support these new style deployments © 2019 IBM Corporation
- 6. 6© 2019 IBM Corporation The need for asynchronous messaging When applications connect directly to each other their availability and scalability is dependent on both applications, and on the quality of the network connection between them © 2019 IBM Corporation As the the network availability and scalability is stretched, some messaging services, such as IBM MQ, can be used to decouple the applications further from the infrastructure Using a messaging service between applications decouples the overall availability and scalability from the applications. Availability is still dependent on the messaging service and the applications’ connectivity to it
- 7. On-premise, software and the MQ Appliance Run MQ yourself in public or private clouds Let IBM host MQ for you with its managed SaaS MQ service in public clouds, IBM Cloud and AWS Run IBM MQ in any location or cloud, exactly as you need it © 2019 IBM Corporation 7 Kubernetes AWS Linux Windows Solaris AIX IBMi IBM Z Appliance zLinuxHPE NonStop Azure AWS Red Hat OpenShift IBM Cloud Private
- 8. For example: Deploy and run MQ in IBM Cloud to suit your needs (Usual requirements for supported operating system and file system) IaaS Open-source containers Vendor container platform IBM MQ on Cloud service Managed container platform BYOL SaaS (PaaS) © 2019 IBM Corporation Docker, Kubernetes, … IBM Cloud Private, Red Hat OpenShift, … IBM Kubernetes Service Virtual machine, Bare metal server software install
- 9. MQ on Cloud service Up and Running in Minutes Managed for You Hourly billing Enabled for Hybrid Cloud Connectivity © 2019 IBM Corporation Available in multiple clouds! A managed service for IBM MQ operated by IBM, so that you can focus on your applications 9 cloud.ibm.com/catalog/services/mq Lite plan - no credit card required!
- 10. Choose your preferred level of responsibility © 2019 IBM Corporation MQ on Cloud service Data centre Networking Servers Storage Hypervisor Virtual machine OS patching MQ patching QM failover Queues, Topics Clustering QM availability/restart Q / Msg monitoring Application System monitoring Data centre Networking Servers Storage Hypervisor Virtual machine OS patching MQ patching QM failover QM availability/restart Queues, Topics Clustering Q / Msg monitoring Application System monitoring Data centre Networking Servers Storage Hypervisor Virtual machine OS patching MQ patching QM failover QM availability/restart Queues, Topics Clustering Q / Msg monitoring Application System monitoring Data centre Networking Servers Storage Hypervisor Virtual machine OS patching MQ patching QM failover QM availability/restart Queues, Topics Clustering Q / Msg monitoring Application System monitoring Data centre Networking Servers Storage Hypervisor Virtual machine OS patching MQ patching QM failover QM availability/restart Queues, Topics Clustering Q / Msg monitoring Application System monitoring Virtual machine IaaS install IBM Cloud Private on IKS IBM Cloud Private IaaS install On-premises software MQ Customer IBM Kubernetes masterKubernetes masterKubernetes master
- 11. Relative location of the components © 2019 IBM Corporation MQ on Cloud service Virtual machine IaaS install IBM Cloud Private on IKS IBM Cloud Private IaaS install Customer account(s) IBM account(s) QM QMQM QM K8s Master K8s Master K8s Master Apps Apps Apps Apps • Connectivity between accounts (public / private backbone depending on the specific case) • Data egress charges (for data out of an account, but not inbound) • (K8s = Kubernetes)
- 12. General architectural best practices for IBM MQ in the cloud
- 13. Architectural best practices for IBM MQ in the cloud 1. Avoiding long distance client connections 2. Deploy application-centric queue managers 3. Use cloud-hosted gateway queue managers 4. Use horizontal scaling for increased availability 5. Connectivity options to on-premises 6. Connecting to other messaging services 7. Deployment isolation to aid organizational structure 8. Use IBM MQ security features where appropriate
- 14. 14© 2019 IBM Corporation Avoid long distance client connections QM_A QM_B On-premises Cloud Not recommended App1 App2 App3 • Good practice is to minimize the distance between applications and the messaging layer • Allows the messaging infrastructure to handle the potential errors, retry and latency so that you don’t have to handle them in application code • Also reduces the network/firewall configuration headaches because the connections are not from individual apps (see later slide) • The improved resilience benefits of using local queue managers typically outweighs the administrative simplicity of keeping all queue managers inside the on-premises data centre • For IBM MQ this means the most benefit is found by placing queue managers in the same location as the applications they serve © 2019 IBM Corporation
- 15. • The flexibility of cloud style deployments makes it easier to deploy a larger number of small QMs, where previously you might have used one big QM • Assigning “one” app to a QM means it is; • Easier to schedule maintenance • Apply finer grained sizing / scaling • QM failure only affects a single application • Continue processing locally if a remote QM is down • Sometimes ownership is devolved to application teams, allowing them to innovate more quickly • Use IBM MQ’s strength in QM-to-QM channels to connect application domains where necessary 15© 2019 IBM Corporation Deploy application-centric queue managers MQ MQ1 MQ2 MQ3 MQ4 App1 App2 App3 App4 App1 App2 App3 App4
- 16. • Ensures that the long distance connectivity is handled by IBM MQ, which is designed to handle the error, retry and latency issues that can occur with remote connections • Reduces the number of cross- location links which makes configuration and administration more straightforward • Gateway QMs might often be managed by a central team, as they are used by many apps 16© 2019 IBM Corporation Use cloud-hosted gateway queue managers On-premises Cloud Not recommended
- 17. 17© 2019 IBM Corporation Use gateway queue managers (2) Cloud #2 Cloud #1 On-premises • Provide gateway queue managers in each domain where applications are deployed, in order to maximise performance and reliability • Either different cloud providers, or different locations/different accounts within a provider • Gateway QMs should be configured to be highly available and scalable (e.g. multiple instances and routes) as they are used by multiple applications
- 18. • Applications with high throughput or availability requirements should be designed to use multiple equivalent queue managers rather than depend upon a single QM • Same benefits as described for having multiple gateway queue managers • Allows individual queue managers to be taken out of service for upgrade (or due to a failure) without affecting the overall service • Add additional queue managers in order to increase the capacity of the system • Requires applications to be written and configured with this pattern in mind (message ordering, multiple endpoints) • IBM MQ product capabilities such as ”uniform clusters”, CCDT, ConnectionNameList and auto client-reconnect can help support these topologies 18© 2019 IBM Corporation Horizontal scaling for increased availability QM1a QM1b QM1c
- 19. • MQ clustering provides the ability to link together queue managers so that they dynamically configure the necessary channels to allow messages to flow where necessary • Also provides workload balancing and availability routing for distribution of messages across multiple equivalent instances of queues • Note that MQ clusters are fully connected, which can increases the number of links between locations – can mitigate this by choosing the scope of the cluster(s) 19© 2019 IBM Corporation Use MQ clustering to provide transparent routing On-premises Cloud
- 20. • How do you intend to connect from on- premises to your cloud location(s)? • Various options using public Internet, often via use of an on-premises “agent” • Direct connection, without agent • Outbound initiation only (not addressable inbound) • Bi-directional initiation (addressable from Internet) • VPN • IBM Cloud Secure Gateway • IBM MQ Internet Pass-thru • Private or telco backbone connections • e.g. Direct Link (IBM), Direct Connect (AWS) • Different pros/cons of each in terms of security, configuration, throughput and cost 20© 2019 IBM Corporation Identify hybrid cloud connectivity approach Direct / public Agent / VPN Dedicated link
- 21. 1. Pre-packaged bridge – IBM Event Streams on Cloud (hosted Apache Kafka, formerly MessageHub) includes a built-in bridge for connecting to IBM MQ – IBM Event Streams (Apache Kafka in IBM Cloud Private) also includes IBM MQ connectors 2. Integration flow – Flow primitives exist in various integration products to allow put/get with MQ to be combined with other providers – e.g. AppConnect Enterprise on Cloud, or on- premises 3. Manual coding – Write custom code to integrate the two providers – Apache Camel provides a Java based framework for integrating providers, including using the JMS interfaces (not endorsed directly by IBM) • In some cases you may be requested to transfer messages between IBM MQ and other messaging providers • Consider carefully the application scenarios for doing this and whether it is appropriate for the solution as a whole • Important considerations: • Error handling and retry of the transfer • Performance and throughput • Resilience and availability • Quality of service requirements 21© 2019 IBM Corporation Connecting to other messaging services
- 22. Technical implications • If you choose to use separate accounts you have a choice whether each account needs a local queue manager • May be affected by whether each account is owned by a central team or delegated to the project / department • Network reliability is less of a concern within the same physical location, but connectivity may still be affected by cross-account security groups or firewall configurations etc Account structure • You might choose to group parts of your cloud deployment into isolated domains to allow segregated administration or billing • There are two main ways to isolate parts of the environment from each other 1. Using cloud provider capabilities to define a subset of resources within a given account 2. Use a different account for each area • Generally the same goals are achievable in both cases, which include; • Security groups, user ACLs for access • Tools to apportion costs within a single account, or aggregate multiple accounts into a single bill 22© 2019 IBM Corporation Deployment isolation Account #2Account #1 ? Cloud region ?
- 23. 23© 2019 IBM Corporation Central vs LoB administration of MQ In traditional on-premises deployments IBM MQ is often managed by a central MQ team on behalf of application teams © 2019 IBM Corporation In some situations ownership of application-specific queue managers might be delegated to the application teams, to enable them to own their updates in a self-service fashion More feasible for application teams that have a better level of skill in MQ Applications with light workload requirements might connect directly to the gateway queue managers Gateway queue managers are used by multiple applications so are also likely to be managed by a central MQ team Central IT / MQ team responsibility Cloud #2Cloud #1 On-premises On-premises Can use Gateway queue managers to remotely administer other connected QMs via MQ Explorer, runmqsc, PCF or REST API
- 24. 24© 2019 IBM Corporation Use IBM MQ security features where appropriate App 1. TLS channels to encrypt data in motion 2. Authenticate connecting application using a client certificate with Mutual TLS 3. Authenticate application with username/password, backed by operating system, LDAP or custom user registry 6. Encrypt individual message content using Advanced Message Security (AMS) feature – see next slide IBM MQ 4. Fine grained authorization of individual applications to specific queues / topics etc 5. Restrict incoming connections based on a policy using channel authentication
- 25. 1. Automatically encrypted by the sending client so that it can only be decrypted by the intended recipient 2. Or encrypted by the queue manager on receipt, for cases where the application deployment cannot be updated Benefits • No application code changes required • Goes beyond TLS channel security, which only protects data in transit between processes • Message data can only be read by the intended receiving application code • Not on the queue by the system administrator • Not on the disk by your infrastructure or cloud provider • Proven, trusted approach to fulfilling compliance requirements such as GDPR, PCI, HIPAA etc IBM MQ: Security for the Cloud MQ Advanced Message Security (AMS) provides the capability to encrypt messages in transit and at rest between sender and receiver. © 2019 IBM Corporation Application B Application A Channels IBM MQ Advanced Queue Manager Queue Manager
- 26. © 2019 IBM Corporation
- 27. • Gateway queue managers to connect between accounts + locations where necessary (does every domain need to talk to all others?) • Application specific queue managers for cloud deployments where there is more than one app per account • Delegate ownership of the app-specific queue managers to the account teams if they have appropriate skill? • Could consider moving to a similar model on-premises if desirable • MQ Cluster(s) where appropriate to aid routing and workload balancing of messages • Choice whether to route between Accounts directly (e.g. #2 -> #3 in diagram), or always route via on- premises 27© 2019 IBM Corporation Potential customer topology “after” (discussion) Account #2Account #1 IBM Cloud region Account #3 Central IT / MQ team responsibility On-premises Pair of QMs at each Gateway for resilience ?
- 28. More information Further details on this topic can be found in the following blog post; https://developer.ibm.com/messaging/2018/05/17/secure-reliable-communication-multi-cloud-deployment-using-ibm-mq/