IBM Message Hub: Cloud-Native Messaging
- 1. InterConnect 2017 HHM-6883 IBM Message Hub: Cloud-Native Messaging Andrew Schofield Event Services, IBM Cloud Paula Ta-Shma IBM Research 1
- 2. 2 Please note IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
- 4. “Kafka adoption was accelerating…with production deployments of Kafka claiming six of the top 10 travel companies, seven of the top 10 global banks, eight of the top 10 insurance companies, and nine of the top 10 US telecom companies." Kafka is: Everywhere Source: http://www.techrepublic.com/article/an-inside-look-at-why-apache-kafka-adoption-is-exploding/
- 5. Kafka is: Built for Scale 0 1 2 3 4 5 Topic Partition 0
- 6. Kafka is: Built for Scale 0 1 2 3 4 5 Topic 6 7Partition 0
- 7. Kafka is: Built for Scale 0 1 2 3 4 5 Topic 6 7 0 1 2 3 0 1 2 3 4 5 Partition 0 Partition 1 Partition 2
- 8. Kafka is: Built for Scale Node 1 Node 2 Node 3
- 9. Kafka is: Built for Scale Node 1 Node 2 Node 3
- 10. Kafka is: Built for Scale Node 1 Node 2 Node 3 Node 4 Node 5
- 11. Kafka is: Built for Scale Node 1 Node 2 Node 3 Node 4 Node 5
- 12. IBM Message Hub MQ Light API AMQP RESTful API HTTP Apache Kafka TCP
- 13. Building: Message Hub + Public Cloud • Message Hub + OpenWhisk • Monitor events received in Message Hub to trigger actions in OpenWhisk • Blog post • Message Hub + Watson IoT • Push events from Watson IoT to Message Hub for use in real-time applications • Documentation • Message Hub + Object Storage • Push events from Message Hub automatically into Object Storage for cost effective long term storage • Documentation
- 14. • Message Hub + MQ • Consume events from MQ using our new bridge to allow other applications and services to consume directly from Message Hub • Documentation • Message Hub + Integration Bus • IIB supports native Kafka consumer and producer nodes, allowing you to produce to and consume from a Message Hub instance • Blog post Building: Message Hub + Hybrid Cloud
- 16. Message Hub + MQ BRIDGE
- 17. Uses an MQI client connection • Supports connection to multi-instance queue managers Offers at-least-once reliability • There may be duplication in some cases, but not loss • Kafka users are used to at-least-once reliability Partitioning into Kafka topics • By CorrelId, GroupId or round-robin 17 MQ Bridge details Integrated into Message Hub
- 18. Message Hub + Object Storage BRIDGE • Take data from a Kafka topic and copies into an Object Storage container • Output format intended to be easy for processing in Apache Spark • Use cases: • Cost-effective, long-term storage of data • Feeding batch analytics systems
- 19. • Decide how to organize the data in Object Storage • according to Kafka offset • according to date • Decide how often to upload new objects • time-based policy • size-based policy 19 Object Storage Bridge details Data organized according to date
- 20. 20 Example Object Storage Bridge configuration
- 21. Scenario: Ingesting IoT data
- 22. How do you deploy this in IBM Cloud? 22 IBM Cloud Internet of Things Platform Message Hub Data Science ExperienceObject Storage Apache Spark BRIDGE
- 23. Demo: the COSMOS Madrid Traffic Use Case • Objective: Respond more quickly and efficiently to real-time traffic problems. Improve customer satisfaction and reduce costs. • Approach: Monitor data from up to 3000 sensors. React by rerouting buses, modifying traffic lights, etc., based upon knowledge derived from historical data
- 24. 24 Application Architecture on Bluemix
- 25. Demo 25
- 26. 26 Why use Message Hub? Run your own Kafka • Deploy your own Kafka cluster • Deploy Zookeeper cluster to support it • Put systems in place to operate the service 24x7 • Keep up-to-date with the open source community • Handle version-to-version migration and updates Use Message Hub • One-click deploy of a fully managed Kafka environment • Additional custom integration points with the IBM Cloud Platform
- 27. 27 Public vs dedicated Message Hub Public Dedicated Cluster tenancy model Multi-tenant Single-tenant Partition limit 100 per tenant 100 per environment Log data per partition 1GB 10GB Throughput quotas Per-tenant None Bridges Object Storage (out) MQ (in) Could be deployed if required Billing Usage-based • Partitions/month • Messages sent/rcvd Subscription
- 28. • Interesting links: • kafka.apache.org • IBM Message Hub • Check out Paula’s blog on the Object Storage Bridge: • https://www.ibm.com/blogs/bluemix/2017/03/end-to-end-iot-data-pipelines- introducing-the-message-hub-object-storage-bridge/ • Try out Message Hub in our lab: • Streaming Analytics in IBM Bluemix Using IBM Message Hub and Apache Spark • Mandalay Bay South, Level 3 - South Seas H • Wed, 22-Mar 10:15 AM - 12:00 PM Message Hub: Next steps
- 29. 29 Notices and disclaimers Copyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.” Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
- 30. 30 Notices and disclaimers continued Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®, Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.