Connecting Applications Everywhere with ActiveMQ
35 likes14,566 views
This document summarizes a presentation given by Rob Davies at the CamelOne 2013 conference in Boston, MA on June 10-11, 2013. The presentation introduced Apache ActiveMQ, an open-source message broker, and discussed its features including messaging protocols, management tools, high availability, and integration with Apache Camel. It also covered challenges of deploying and maintaining large ActiveMQ clusters and how Red Hat Fuse Fabric can help address these challenges.
![CamelOne 2013
CamelOne
63
USB
7-12 v
3 v GRD 5 v Analog Input Pins
Digital Input/Output Pins
Pins with ~ are PWM
[Analog Output]
GRD
Transmitter/Receiver
Serial Connection
Microcontroller ATmega328
OperatingVoltage 5V
InputVoltage (recommended)7-12V
InputVoltage (limits)6-20V
Digital I/O Pins 14
(of which 6 provide PWM output)
Analog Input Pins 6
DC Current per I/O Pin 40 mA
DC Current for 3.3V Pin 50 mA](https://image.slidesharecdn.com/connectingapplicationseverywherewithactivemq-130611101811-phpapp02/85/Connecting-Applications-Everywhere-with-ActiveMQ-63-320.jpg)
Connecting Applications Everywhere with ActiveMQ
- 1. CamelOne 2013 June 10-11 2013 Boston, MA 1 Connecting Applications Everywhere with ActiveMQ Rob Davies Technical Director, Fuse Engineering, Red Hat Inc.
- 2. CamelOne 2013 CamelOne 2 Rob Davies Technical Director, Red Hat - h"p://www.redhat.com ■ Previously CTO of FuseSource - #1 OS vendor for integration and messaging ■ Software projects: ■ Apache ActiveMQ, ■ Apache Camel ■ Apache ServiceMix ■ On Expert Group for JSR 343: JMS 2.0 ■ Co-‐author of AcCveMQ in AcCon:
- 3. CamelOne 2013 CamelOne Why use Message- Oriented Middleware? Robustness to change Time Independence LocaCon Independence Hide Latency Scalability Event driven Simplicity Configurable Quality of Service PlaRorm and Language IntegraCon Fault tolerant
- 4. CamelOne 2013 CamelOne 4 What is Apache ActiveMQ ? ■ Top Level Apache Software Foundation Project ■ Wildly popular, high performance, reliable message broker ■ Connects to nearly everything ■ Native Java, C/C++, .Net, ■ AMQP 1.0, MQTT 3.1, STOMP (1.0-1.2) and OpenWire ■ STOMP enables Ruby, JS, Perl, Python, PHP, ActionScript … ■ Embedded and standalone deployment options
- 5. CamelOne 2013 CamelOne Messaging: The Basics
- 6. CamelOne 2013 CamelOne Message Channels and Routing • Message Channels • Named communication between interested parties • JMS calls them ‘Destinations’ • Can “tune-in” to multiple channels using wildcards • Can fine-tune message consumption with selectors • Can route a message based on content
- 7. CamelOne 2013 CamelOne Message Channels = JMS Destinations Producer Broker Consumer Consumer Consumer Destination WIDGET Destination ORDER
- 8. CamelOne 2013 CamelOne Point-toPoint Channel: JMS Queues Producer Consumer Consumer Consumer Queue
- 9. CamelOne 2013 CamelOne Publish/Subscribe Channel: JMS Topics Producer Consumer Consumer Consumer Topic
- 10. CamelOne 2013 CamelOne Message Routing : Selectors DestinationProducer Consumer color='blue' Consumer color='red'
- 11. CamelOne 2013 CamelOne Message Routing : Destination Wildcards Topic: BAR.BEER Consumer topic:BAR.> Topic: BAR.WINE Producer
- 13. CamelOne 2013 CamelOne Message Groups Like Parallel Exclusive Consumers ■ Guaranteed ordering of related messages across a Queue ■ But – load balancing of messages across multiple consumers ■ All messages with the same JMSXGroupID go to the same consumer ■ How you group messages is down to the application’s producer ■ To explicitly close a group, set the JMSXGroupSeq to -1 13
- 14. CamelOne 2013 CamelOne 14 Message Groups code: Message message = session.createTextMessage("<foo>hi from Devoxx</foo>"); message.setStringProperty("JMSXGroupID", ”RHT_NYSE"); producer.send(message); Message message = session.createTextMessage(foobye from Devoxx/foo); message.setStringProperty(JMSXGroupID, ”RHT_NYSE); message.setStringProperty(JMSXGroupSeq, -1); producer.send(message); //Starting a Group … //Close a Group …
- 15. CamelOne 2013 CamelOne Deploying ActiveMQ ActiveMQ Can run standalone or embedded ■ As a standalone, or part of a highly available message broker cluster ■ Embedded – easy to use an entire broker in JUnit tests (no need to Mock) ■ In Tomcat, deployed as a war ■ In JEE Server – either co-locate – or use client with JCA ■ ActiveMQ distributions contain a rar for this purpose 15
- 16. CamelOne 2013 CamelOne ActiveMQ: Message Storage
- 17. CamelOne 2013 CamelOne Message Delivery Mode • Messages can be persisted – allowing for decoupled applications • For Queues, any message delivered as PERSISTENT must be stored on long term storage before being delivered to a consumer • For Topics, a message delivered as PERSISTENT will only be stored if there exists a durable subscriber • NON-PERSISTENT messages may also be stored on disk if the memory limits of the broker have been reached 17
- 18. CamelOne 2013 CamelOne Message stores supported by ActiveMQ ■ SQL (JDBC) ■ SQL (JDBC) with journal ■ AMQ Message Store ■ Kaha ■ KahaDB ■ LevelDB 18 Slow but popular Don’t use (deprecated) The “current” default The “best yet”
- 19. CamelOne 2013 CamelOne LevelDB Store vs KahaDB • Fewer index entries per message than KahaDB • Faster recovery when a broker restarts • LevelDB index out-perform Btree index at sequential access . • LevelDB indexes support concurrent read access. • Pauseless data log file garbage collection cycles. • Fewer IOPS to load stored messages. • It exposes it's status via JMX for monitoring
- 20. CamelOne 2013 CamelOne ActiveMQ: LevelDB Store
- 22. CamelOne 2013 CamelOne OpenWire Advantages: • Fast – optimized for ActiveMQ • client failover • automatic reconnect • Client load balancing • Flow control • Many advanced features 22 Disadvantages: • Not a recognized standard • Only Java, C/C++/.Net http://activemq.apache.org/openwire.html
- 23. CamelOne 2013 CamelOne MQTT http://mqtt.org Advantages: • M2M/”Internet of Things” transport • Proposed as an OASIS standard • Extremely light weight • Growing support from vendors and OS products • WebSphereMQ • ActiveMQ + Apollo • Mosquitto • RabbitMQ 23 Disadvantages: • 3.1 does not support Queues • Advanced features not standard • Flow control • Failover etc.
- 24. CamelOne 2013 CamelOne AMQP – see www.amqp.org Advantages: • AMQP 1.0 OASIS standard • Proposed as an OASIS standard • Commoditizes the Broker 24 Disadvantages: • One size doesn’t really fit all • Currently no plans for IBM or Tibco to adopt it
- 25. CamelOne 2013 CamelOne STOMP – http://stomp.github.com Advantages: • Easy to use text based protocol • Can use telnet as a client • Defacto standard: • ActiveMQ + Apollo • HornetQ • RabbitMQ • PocoMQ • StompServer • OpenMQ • Many more … 25 Disadvantages: • Not as fast as binary formats
- 26. CamelOne 2013 CamelOne But there’s more: WebSockets STOMP is a natural wire protocol for WebSockets 26 ActiveMQ Client (Browser) WebSocket Connector Stomp Converter ActiveMQ (Backend) TCP (WebSocket) HTTP(S) Request/Response 1. HTTP Request (Upgrade) 2. HTTP Response 4. Message received from Topic 5. Send WebSocket Data 3. Stomp Subscribe
- 28. CamelOne 2013 CamelOne Command line: • Activemq • activemq-admin • activemq-admin stop • activemq-admin query • activemq-admin bstat • activemq-admin browse • activemq-admin list 28 Runs the broker Manages the broker
- 32. CamelOne 2013 CamelOne High Availability ActiveMQ Cluster ActiveMQ Client Supported by both Java and C++ OpenWire clients
- 33. CamelOne 2013 CamelOne Broker Networks (Store and Forward)
- 34. CamelOne 2013 CamelOne Network of Brokers : Geographically Dispersed
- 35. CamelOne 2013 CamelOne Network of Brokers : Network with Master/Slave
- 36. CamelOne 2013 CamelOne Scaling Networks of Brokers • Brokers share routing information across networks • All destinations are considered Global • This is really convenient • Though it starts to get problematic with 1000’s of brokers • However – we can do filtering to shape the traffic across networks • But this involves a lot of manual configuration 36 crikey!
- 37. CamelOne 2013 CamelOne • Brokers share routing information across networks • All destinations are considered Global • This is really convenient • Though it starts to get problematic with 1000’s of brokers • However – we can do filtering to shape the traffic across networks • But this involves a lot of manual configuration 37 Time to talk about Apache Camel … Scaling Networks of Brokers
- 38. CamelOne 2013 CamelOne ActiveMQ with embedded Camel Service Consumer Consumer ActiveMQ Broker Flexible and is faster J
- 39. CamelOne 2013 CamelOne 39 ActiveMQ with embedded Camel beans broker brokerName=testBroker xmlns=http://activemq.apache.org/schema/core transportConnectors transportConnector uri=tcp://localhost:61616/ /transportConnectors /broker import resource=camel.xml/ /beans Import Camel into ActiveMQ broker config:
- 40. CamelOne 2013 CamelOne 40 Setup Camel Context in the usual way beans … camelContext id=camel xmlns=http://camel.apache.org/schema/spring route descriptionExample Camel Route/description from uri=activemq:topic:audio/ setHeader headerName=JMS_AMQP_MESSAGE_FORMAT constant0/constant /setHeader to uri=activemq:queue:audio/ /route /camelContext … /beans
- 41. CamelOne 2013 CamelOne Scaling Networks – use Apache Camel • Allows for non-chatty networks to be established • Routing information can be externalized to the broker • Successfully used in production for 1000’s of brokers today 41
- 42. CamelOne 2013 CamelOne More problems with Large Deployments
- 43. CamelOne 2013 CamelOne Problems – Deploying and maintenance • Main problems • Installing brokers on multiple hosts • ssh, untar, set directories and environment • Setting configuration manually for every broker • copying xml config, tweaking, testing • Updating configuration across cluster • Upgrading brokers It’s a tedious and error-prone process
- 44. CamelOne 2013 CamelOne Problems – Traditional best- practice tips • Keep XML as a template and configure node-specific details through properties • Keep configuration in SVC system (git, svn, ...) • Keep configuration separate from installation for easier upgrades Deployment with Fuse Fabric moves it to the next level
- 45. CamelOne 2013 CamelOne Problems - Clients • Topology is very “static” • Clients need to be aware of topology • Clients need to know broker locations • Changes are not easy as clients need to be updated • Adding new resources (brokers) requires client updates • Not suitable for “cloud” deployments Fuse Fabric makes deployments more “elastic”
- 46. CamelOne 2013 CamelOne Fuse Fabric to the rescue!
- 47. CamelOne 2013 CamelOne Fuse Fabric – Key Features Fuse IDE hawCo ActiveMQ Cluster Camel Endpoints ServiceMix Runtime Registry • Supports Hybrid deployments • Distributed Configuration • Runtime registry • Centralized Management
- 48. CamelOne 2013 CamelOne FuseMQ features • mq-base profile • Defines OSGi features and bundles to be installed • Defines basic broker settings • mq-create command • Helper command for creating brokers • Creates an new profile based on mq-base • Optionally creates new containers • Assigns the profile to containers (essentially starts the broker) FuseMQ = ActiveMQ deployed in Apache Karaf +
- 49. CamelOne 2013 CamelOne Why is Integration and Messaging Important ? Head Office Enterprises Need to know Everything!
- 50. CamelOne 2013 CamelOne There are more things to Integrate with!
- 51. CamelOne 2013 CamelOne The Internet of Things • Uniquely identifiable objects and their virtual representations in an internet-like structure • Originally defined by Kevin Ashton, co-founder of Auto-ID center at MIT • Today its meaning has evolved to encompass a world where physical objects are integrated into the information network, and where physical objects participate in business processes. 51
- 52. CamelOne 2013 CamelOne Machine to Machine • M2M involves collection and transmission of event level data (used to be called telemetry) from intelligent devices • Machines can interact over the net – and/or attached networks – by wired and wireless networks • Payloads are typically minimal (temperature, pressure, location, metering etc.) 52
- 53. CamelOne 2013 CamelOne Internet of Things Time Connected Devices on the internet 2010 2000 2020 1 Billion 50 Billion 10 Billion Ubiquitous posiConing – locaCng people and everyday objects RFID tags for Inventory control Advanced sensors, Enterprises fully connected: integraCon of everything Geo-‐located devices Microcontrollers internet enabled Vending machines, lone workers, POS, vehicles, planes, tolls, power systems, transport, telemedicine, home appliances, power systems, etc.
- 54. CamelOne 2013 CamelOne M2M Examples • Smart Energy – Smart Grid uses smart meters for monitoring energy uses by premises for billing, substation and transmission line monitoring, energy production (renewables) • Inventory Control – smart labels and RFID tags – connected scanners pass information upstream to servers for monitoring • In-Vehicle Systems Maintenance information, global positioning, ‘black-box’ (speed and driving habits), wireless payments for tolls/gas • Environmental Monitoring Weather stations (temperature, air-pressure), ocean monitors, earth movements, volcanic activity crop yields etc. • Livestock monitoring Cow herds, monitored for optimum time for milking, sheep herds for location 54
- 55. CamelOne 2013 CamelOne M2M Topology 55 Gateway Hadoop Servers Analytics Gateway Internet Mesh Network Mobile wireless
- 56. CamelOne 2013 CamelOne Power Consumption! • Battery life for mobiles, power consumption for smart devices is an important consideration. • Facebook use MQTT for real-time updates – efficiency was one of the things considered • MQTT is more efficient at establishing connections, sends information more reliably, and consumes less power to transmit data than HTTP. 56
- 57. CamelOne 2013 CamelOne Deluge of Data • Billions of devices send information will require storage and processing of terabytes or even petabytes of data. • Big Data Ingestion: Processing of vast amounts of data business generates to make good decisions is only half the problem. Ingesting that data from all areas of the enterprise will require TBs of data to be ingested in highly reliable and scalable way. 57
- 58. CamelOne 2013 CamelOne Need an Eco system of technologies Dev • Integration is getting harder • Traditional message brokers are key to connect “everything” • Multiple protocols need to be supported • Camel can provide integration at the edges – and the data center • Technologies like Fuse Fabric provide centralized provisioning and management • Need to mix protocols, brokers and “message routers” to scale FuseMQ Clusters Arrival Airport 1
- 60. CamelOne 2013 CamelOne Messaging through the Internet of Things ActiveMQ AMQP/MQTT routers AMQP/MQTT routers AMQP/MQTT routers MQTT MQTT-‐S AMQP AMQP AMQP MQTT MQTT websockets
- 61. CamelOne 2013 CamelOne Simplified demo Java MQTT Client AMQP MQTT ActiveMQ MQTT/AMQP Gateway ActiveMQ AMQP/OpenWire Broker JMS OpenWire Client
- 62. CamelOne 2013 CamelOne What is Arduino ? • Open Source Hardware microcontroller • Cheap and easily available. • Open Source Software. • Widespread: many projects. • Extra HW (shields) available (e.g. WiFi shield)
- 63. CamelOne 2013 CamelOne 63 USB 7-12 v 3 v GRD 5 v Analog Input Pins Digital Input/Output Pins Pins with ~ are PWM [Analog Output] GRD Transmitter/Receiver Serial Connection Microcontroller ATmega328 OperatingVoltage 5V InputVoltage (recommended)7-12V InputVoltage (limits)6-20V Digital I/O Pins 14 (of which 6 provide PWM output) Analog Input Pins 6 DC Current per I/O Pin 40 mA DC Current for 3.3V Pin 50 mA
- 64. CamelOne 2013 CamelOne Arduino Programming • 3 types of memory on Arduino • Flash memory (program space) – 32 Kb • SRAM – used by sketches – 2 Kb • EPROM – long term memory – 1 Kb • Programming language based on Wiring: C/C++ library designed to make input/output easier. Programs are called sketches. • Arduino has a simple IDE, available on Windows, Linux and Mac
- 65. CamelOne 2013 CamelOne Arduino MQTT to WebSockets Arduino AMQP MQTT ActiveMQ MQTT/AMQP Gateway ActiveMQ AMQP/WebSockets Broker HTML 5/WebSockets
- 66. CamelOne 2013 CamelOne Useful Resources: • http://activemq.apache.org • http://camel.apache.org • http://fuse.fusesource.org/fabric/docs/overview.html • http://fuse.fusesource.org/mq/docs/mq-fabric.html • https://github.com/fusesource/fuseide • http://hawt.io • http://www.arduino.cc/ Messaging and Integration Developer tooling Management Configuration and provisioning Arduino