Building scalable flexible messaging systems using qpid
- 1. Building Scalable Messaging Systems with Qpid Lessons Learned from PayPal
- 2. Background @ PayPal • Handles 60% of all web transacBons • One of the largest Oracle instances • Mix of proprietary systems • Mix of 1000’s of stateless processes • TradiBonal JEE applicaBons • Payments are generally asynchronous • Payments are generally messages
- 3. Basic Requirements • Scaling – Highly Scalable – ParBBonable – Cloud Friendly • Failure – ConBnuously Available – No Avoid Single Point of Failure – Nothing Shared • Latency – Near Real Time Hub
- 4. Why • Desired an open messaging protocol • Cross plaSorm interoperability (C++, Java, Python) • Required very low latency • Eventual interoperability with AcBveMQ • Ability to influence the community
- 5. Where We Started • Simple Network of Brokers • Load Balanced via L5 Switch • Round Robin, Least/Min Rule • Replicated Point to Point Consumer AQP Client Consumer AMQP Client AMQP Provider AMQP Provider F5 LB
- 6. What We Found • Scale – 20 billion 2K Messages Per Day – VariaBon Message Size > Latency • ConnecBons – Short lived processes strain the broker – @5000-‐6500 broker begins to flail • RouBng Concerns – DistribuBng connecBons – Prohibited InVM messaging
- 7. Next EvoluBon • Create DisBnct Layers of Brokers – Front Tier – Mid Tier – Core Tier • ParBBon Each Layer By FuncBon or Actor – User Type (Consumers, Merchants, API) – Business FuncBon (Risk, Payments, Account Servicing) – System FuncBon (Events, Services, Logging) – Cloud Friendly • Isolate ParBBons within the Broker Consumer AMQP Provider Consumer AMQP Provider AMQP Provider AMQP Provider AMQP Provider Service AMQP Provider Service
- 8. Interfaces • FederaBon SemanBcs Part of the Address • Externalize Addressing • Use “pure” AMQP or JMS wherever possible AMQP JCA JMS Spring PayPal JEE Camel ApacheCXF SASL JAAS Kerberos PayPal Other AMQP Provider
- 9. Network Local Local Local Gateway Gateway Remote Remote Infra Infra
- 10. FederaBon • DisBnct Request & Response Exchanges • Local & Remote Request DesBnaBons – qpid-config -a localhost:5673 add exchange direct service_gateway.direct! – qpid-config -a localhost:5674 add exchange direct service_impl.direct! – qpid-config -a localhost:5673 add queue user.lifecycle.request! – qpid-config -a localhost:5674 add queue user.lifecycle.request.impl! • Requests use queue routes – qpid-route queue add localhost:5674 localhost:5673 service_impl.direct user.lifecycle.request! • Responses use dynamic routes – qpid-route -v dynamic add localhost:5674 localhost:5673 service_res.direct – durable! • Responses use unique binding addresses for rouBng
- 11. FEDERATION Example Gateway (5673) Remote (5674) Remote (5674)
- 12. Client/Publisher • Message size is important • Avoid default reply to implementaBons • Pull configuraBon versus pre-‐packaging
- 13. Managing the Config Config Puppet Package Config ZooKeeper Qpid Qpid Qpid
- 14. Monitoring • Qpid Management Framework – Each Object In the Broker Publishes Event – Events are Messages (Topic) that are Routable • Describe Interest In Events • Listeners that subscribe & dispatch – SNMP – Nagios – Internal Logging
- 15. What Can Be • Agent • Binding • Bridge • Broker ConnecBon • Exchange • Link • Queue • SubscripBon • System • VHost • Events – New Objects – Updates – Failures (Links) • ConfiguraBon – ProperBes – States • StaBsBcs – TXN – Messages – Latency
- 17. Performance 17 Configura7on 1 100 1024 2048 4096 8192 16384 32768 65356 130712 Single Node TCP over 1gbE 0.6 0.6 0.66 0.64 0.65 0.66 0.73 0.9 1.07 1.55 MulB Node TCP over 1gbE 0.9 0.9 0.98 1.08 1.13 1.25 1.49 2.14 3.35 5.72 MulB Node RDMA over IB 0.42 0.43 0.43 0.44 0.44 0.45 0.51 0.55 0.63 0.74 Raw QPID average roundtrip message Bmes in milliseconds over 100K messages. Use case comparison between MBSI architectures by Infrastructure. Times are average per message over 1K messages Throughput: • Single Node 141K Messages Per Second @ 1K Message Size & Single Producer • I/O Bound on the Producer Use Case Binary (1024) XML (1024) Binary (16384) XML (16384) Binary (65536) XML (65536) Binary (131072) XML (131072) QPID/SSL (Sync) 2.66 3.33 6.11 7.17 11.73 16.07 19.71 27.86 QPID/SSL (Fire-‐and-‐ forget/Non-‐Ack'd) 0.51 0.56 0.54 0.74 0.8 1.44 2.05 2.59 QPID/SSL (Fire-‐and-‐ forget/Ack'd) 1.04 1.08 3.52 2.04 3.69 4.08 5.84
- 18. How We Can Use It • PayPal Cloud – Openstack integraBon via AMQP – Dynamic Scaling via QMF events • Possible ApplicaBons – Mobile via JavaScript Proton – Payment Devices
- 19. OpportuniBes • AMQP Links between Qpid & AcBveMQ – Heterogeneous messaging fabric • Embedded Messaging Engines – In car devices – Point of Sale – Phones • Replace proprietary service frameworks with Proton • Replace Qpid Libraries with Proton