JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel
Download as PPTX, PDF1 like1,501 views
The document discusses scalable messaging solutions using JBoss A-MQ and Apache Camel, highlighting their features, protocols, and scalability options tailored for large retail industries. It emphasizes the importance of vertical and horizontal scalability to manage increasing data loads and improve integration processes. Additionally, it covers the architecture, message stores, and examples of configuration to optimize performance in a messaging environment.
![MQTT
MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol.
URI- mqtt://Host:[Port]?transportOptions
13
Facebook Messenger Smart Santander Features
Open
Simple
Works well in
constrained networks](https://image.slidesharecdn.com/judcon2014scalablemessagingwithjbossa-mq14-140322061332-phpapp01/85/JUDCon2014-ScalableMessagingWithJBossA-MQ-and-Apache-Camel-13-320.jpg)
![AMQP
Features
Multi-Vendor
interoperability
Highly reliable
Feature rich
AMQP (Advanced Message Queueing Protocol) is an openly published wire
specification for asynchronous messaging
URI: “amqp:// Host:[Port]?transportOptions
14](https://image.slidesharecdn.com/judcon2014scalablemessagingwithjbossa-mq14-140322061332-phpapp01/85/JUDCon2014-ScalableMessagingWithJBossA-MQ-and-Apache-Camel-14-320.jpg)
JUDCon2014-ScalableMessagingWithJBossA-MQ and Apache Camel
- 1. 1
- 2. Scalable messaging with JBoss A-MQ and Camel Bhavani Ananth @bhavani_an in.linkedin.com/in/bhavaniananth/ 2 Naveen Raj Balasubramaniam @Naveenrajbala in.linkedin.com/in/naveenraj/
- 3. Messaging And Integration JBoss A-MQ JBoss A-MQ – Protocols JBoss A-MQ – Message Stores Vertical and Horizontal Scalability 3 Camel in brief! Agenda
- 5. Scalable Solution 5 Large Retail Industry Billing Go Online Vendor Inventory Warehouse Logistics Integration Infrastructure MessagingMessaging Messaging Messaging MessagingMessaging
- 6. Data Is Growing Every Second – Data Deluge More Data Due to Emergence of Internet Of Things Need For A Scalable Solution
- 7. What do we have! Red Hat JBoss A-MQ 7
- 8. JBoss A-MQ 8
- 9. What is JBoss A-MQ 9 JMS 1.1-compliant messaging system based on Apache Active MQ. Consists of a broker and client-side libraries enabling remote communication among distributed client applications.
- 10. Architecture and Features 10 JBoss-AMQ Apache Karaf + Fuse Fabric Monitoring and Management Apache Active MQ STOMP MQTT Open Wire AMQP NMS JMS
- 11. JBoss A-MQ – Protocols 11
- 12. Protocols Supported 12 Client API’s JMS NMS CMS Rest/Ajax Wire Protocols MQTT AMQP STOMP Open Wire Transports TCP/SSL HTTP/HTTPS WS/WSS
- 13. MQTT MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. URI- mqtt://Host:[Port]?transportOptions 13 Facebook Messenger Smart Santander Features Open Simple Works well in constrained networks
- 14. AMQP Features Multi-Vendor interoperability Highly reliable Feature rich AMQP (Advanced Message Queueing Protocol) is an openly published wire specification for asynchronous messaging URI: “amqp:// Host:[Port]?transportOptions 14
- 15. STOMP STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol. URI: “stomp://localhost:61613” 15 Broker Producer STOMP Consumer (Ruby,PHP) TCP Features Simple Used in scripting languages
- 16. Openwire Features Default wire format used by ActiveMQ. Efficient Supports auto reconnect Supports client failover OpenWire is the cross language Wire Protocol to allow native access to ActiveMQ from a number of different languages and platforms 16
- 17. JBoss A-MQ – Message Stores 17
- 18. Message Stores KahaDB message store Distributed KahaDB message store LevelDB message store – the best in terms of scalability and performance Journaled JDBC adapter Non-journaled JDBC adapter 18
- 19. Message Broker Message Cursors Message Store and Message cursors in an Retail Infrastructure 19 Vendor Systems System A System B Inventory System Message Store App Queue 2 Dispatch queue Pending Cursor App Queue 1 App Queue 3 Queue impl details
- 20. Message Broker Message Cursors – Increase Scalability of Message Stores Store Based cursors for fast Consumers (Retail Scenario) 20 Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
- 21. Store Based cursors for slow Consumers Message Cursors – Increase Scalability of Message Stores Contd… 21 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
- 22. VMCursors Message Cursors – Increase Scalability of Message Stores Contd… <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb"> <pendingQueuePolicy> <vmQueueCursor/> </pendingQueuePolicy> </policyEntry> 22 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages Vendor Systems System A System B
- 23. File Based Cursors •Does not scale well if the consumers are behind by a large margin •Used for non persistent messages Message Cursors – Increase Scalability of Message Stores Contd… 23 Message Broker Dispatch Queue Pending Cursor Message Store Inbound Messages File Based Cursors Temporary Files Vendor Systems System A System B
- 25. Vertical Scalability Capacity of a single broker to support large numbers of connections from consumers and producers. To Consider Thread Count CPU Usage Memory Usage 25
- 26. Vertical Scalability – Reducing Thread Count NIO protocol <transportConnectors> <transportConnector name="nio" uri="nio://0.0.0.0:61616"/> </<transportConnectors> 26
- 27. JBoss A-MQ – Vertical Scalability – Reducing Thread Count Disable Dedicated Task Runner ACTIVEMQ_OPTS="-org.apache.activemq.UseDedicatedTaskRunner=false“ Use optimized dispatch for queues <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" optimizedDispatch="true"> ... </policyEntry> </policyEntries> </policyMap> </destinationPolicy> 27
- 28. Vertical Scalability – CPU Enable tightEncodingEnabled uri = "failover://(tcp://localhost:61616wireFormat.tightEncodingEnabled=false)"; Enabling compression tcp://hostA:61617?jms.useCompression=true 28
- 29. Vertical Scalability – Memory Allocate more memory to broker • pass the –Xmx option to the JVM. ACTIVEMQ_OPTS="-Xmx2048M -Dorg.apache.activemq.UseDedicatedTaskRunner=false“ Configure big enough memory usage <systemUsage> <memoryUsage> <memoryUsage limit="1024 mb" /> </memoryUsage> </systemUsage> 29
- 30. Horizontal Scalability – Network of Brokers Concepts Network Connector Transport Connector Store and Forward 30 Broker-1 Broker-2 Network Connector Producer Transport Connector Consumer Transport Connector
- 31. Retail Example – Hub and Spoke Topology 31
- 32. Store and Forward- Configuration Store Store Retail Outlet Ware house Fire wall <networkConnectors> <networkConnector uri="static:(tcp://warehouse:61617)" userName="admin" password="admin“ duplex=“true” conduitSubscriptions=“true” decreaseNetworkConsumerPriorit y=“false” /> </networkConnectors>
- 33. When there are a lot of producers Brokers, A, B, and C, accept connections from message producers. Brokers, X and Y, accept connections from message consumers. Concentrator Topology - Producer Load Balancing 33 A B C YX C1 C2 C3 C4
- 34. Some Other Topologies 34 Tree Topology Graph Topology Mesh Topology
- 35. Retail Example-High Availability Shared Store Retail Master Broker Retail Slave Broker Wareh ouse Slave Broker Wareh ouse Master Broker Shared Store Network Connection on Failure Network Connection <networkConnectors> <networkConnector uri="static:(tcp://retail- master:61617, tcp://retail- slave:61617)" </networkConnectors> <services> <masterConnector remoteURI=“tcp://retail- slave:62001” userName=“admin” password=“admin”/> </services>
- 36. Other Master Slave Topologies Master Broker Slave Broker Store Store Message Shared Nothing Master Slave Reliable Only one slave possible Downtime when master fails Client 36
- 37. Shared Storage Master-Slave Master Broker with file lock Slave Broker Shared File System or SAN Shared File system Master Slave Recommended if SAN is available Easy to configure No restriction on the slaves Client Slave Broker Slave polls for lock 37
- 38. Hybrid Scaling 38 Client Acc Payable Master Acc Payable Slave Accounts Payable Information Acc Receivabl e Master Acc Receivabl e Slave Accounts Receivable Information Client Complexity Less broker cross talk
- 39. Apache Camel A ride to solve your integration problems 39
- 40. Camel-What is it Open source integration framework Based on Enterprise Integration Patterns(EIPs) Supports more than 120+ components Supports DSL programming with Spring, Java, Scala and groovy Flexible deployment Can run standalone with maven 40 CBR Wire Tap TransformationEndpoint Translator
- 42. Camel Coding Structure import org.apache.camel.builder.RouteBuilder; public class MyRouteBuilder extends RouteBuilder { public void configure() { from("file:SourceFolder?noop=true") .to("file:TargetFolder"); } } <camelContext xmlns="http://camel.apache.org/schema/spring"> <route id=“FileToFile"> <from uri="file:SourceFolder?noop=true""/> <to uri="mock:inside"/> </route> </camelContext> Java DSL Spring DSL 42 Read a file from SourceFolder and write it to TargetFolder
- 43. 43 <route> <from uri="activemq:JUDCon?destination.consumer.exclusive=true&destinati on.consumer.prefetchSize=50"/> <to uri="mock:2014"/> </route> Routes Camel and Messaging Configuring the Destination Options at the Endpoint in the Routes
- 44. Camel and Messaging Contd… <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory"> <property name="brokerURL" value="tcp://localhost:61616" /> </bean> Beans 44 <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent"> <property name="configuration" ref="jmsConfig"/> <property name="transacted" value="true"/> <property name="cacheLevelName" value="CACHE_CONSUMER" /> </bean> <bean id="pooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop"> <property name= "maxConnections" value="8" /> <property name="connectionFactory" ref="jmsConnectionFactory" /> </bean> <bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration"> <property name="connectionFactory" ref="pooledConnectionFactory"/> <property name="concurrentConsumers" value="10“ /> </bean>
- 45. Camel – Scalability Techniques 45 Throttle Load Balancing from("activemq:queueA").throttle(10). to("activemq:queueB") <from uri=" activemq:queueA " /> <throttle maximumRequestsPerPeriod="3“> <to uri=" activemq:queueB " /> from(" activemq:queueA "). loadBalance().roundRobin(). to(" activemq:queueB ", " activemq:queueC", " activemq:queueD"); <from uri=" activemq:queueA "/> <loadBalance> <roundRobin/> <to uri=" activemq:queueB"/> <to uri=" activemq:queueC "/> <to uri=" activemq:queueD "/> </loadBalance>
- 46. 46 Embedding Camel into Your Broker • Reduces serialization cost • Reduced networkWhy? • Using Interceptors to your brokerWhen? • Add the camel route to activemq.xmlHow?
- 47. Summary • Messaging and Integration • Significance of Asynchronous messaging • How scalability is important to solve large flow of data • Scalability – Vertical and Horizontal • What's camel and Integration 47
- 48. Useful Links References • http://camel.apache.org/architecture.html • http://camel.apache.org/jms.html • http://activemq.apache.org/broker-camel-component.html 48
- 49. Thank You Scale Your Enterprise ECO System 49 @bhavani_an @Naveenrajbala