Distributed Crypto-Currency Trading with Apache Pulsar
2 likes1,632 views
The document outlines a company's initiative to automate a crypto-currency arbitrage discovery platform that leverages Apache Pulsar to facilitate trading across geographically distributed desks while complying with financial regulations. Key requirements include geo-replication of trade data from over 50 exchanges, sub-second message latency, and ensuring net-zero positions on trades. The architecture aims for real-time data processing, high throughput, and low latency to effectively identify and exploit market inefficiencies.
Distributed Crypto-Currency Trading with Apache Pulsar
- 1. Distributed Crypto-Currency Trading with Apache Pulsar David Kjerrumgaard Director of Solution Architecture
- 2. âą Company wishes to automate a crypto-currency arbitrage discovery platform in order to exploit the spreads between crypto-currency prices offered by brokers across multiple crypto-currency exchanges across the globe. âą Currently, the company has multiple geographically distributed trading desks (North America, Europe, and Asia). Due to ïŹnancial regulations, each trading desk has accounts within its geographical region only and cannot execute trades outside of its region. âą To overcome this restriction, the solution must provide the capability to perform a trade in one region on behalf of a user in another region. âą In order to minimize risk, the company does not want to retain any open (long or short) positions. Therefore all arbitrage trades must result in a net zero position, i.e. every trade must be cleared. The Business Use Case
- 3. âą In order to accommodate this, the platform will need to support: âą Geo-replication of trade & quote information ingested from 50+ exchanges. âą The ability to send large, crypto-currency payloads between trading desks in order to successfully complete the purchase of a security in one region and the selling of the same security in another. âą Since we are sending actual crypto-currency payloads security and zero-message loss is a MUST. âą Given the volatility of the crypto-currency prices, sub-second message latency is critical to identify and exploit any market inefïŹciencies. All opportunities must be identiïŹed, acted upon, and trades completed before the price moves. Failure to do so could result in trading losses. Business Use Case Requirements
- 4. ⹠Intelligent platform for fast-moving data ⹠Built with open source technology proven at scale at Twitter, Yahoo, Salesforce⊠⹠Founded by a team of data processing veterans from Twitter, Yahoo, Google, ⊠What is Streamlio
- 5. Interfaces APIs Libraries & Connectivity Intelligent platform for fast data Real-time processing Messaging & queuing Stream storage ConnectorsClientData SourceStormKafka Functional Management Resource Management Metadata Security Monitoring Management UI Powered by Powered by Powered by
- 6. Apache Pulsar fast, durable, flexible pub/sub messaging
- 7. What is Apache Pulsar? 7 Ordering Guaranteed ordering Multi-tenancy A single cluster can support many tenants and use cases High throughput Can reach 1.8 M messages/s in a single partition Durability Data replicated and synced to disk Geo-replication Out of box support for geographically distributed applications Deployed globally in 10+ data centers with full replication and has processed more than 100 trillion messages to date. Delivery Guarantees At least once, at most once and effectively once Low Latency Low publish latency of 5ms at 99 pct Highly scalable Can support millions of topics
- 8. âą Each of the three trading desks ingests real-time trade and quote information from all of the exchanges within their respective regions and publish them into a shared, geo-replicated Pulsar topic. âą Trade & Quote information published in one region is available in another âą Apache Pulsar provides built-in WebSocket clients for all of its topics, which are used to drive the trading dashboards used by the traders. âą When an arbitrage opportunity is identiïŹed, it will be published to the arbitrage topic for further evaluation. The Solution Architecture
- 12. âą Subscribes to the Cryptocompare API. âą You can subscribe to multiple feeds in a single request, but API throttles requests to 1000 / minute. âą Subscription requests have the following format: âą {FeedType}~{ExchangeName}~{FromCurrencySymbol}~{ToCurrencySymbol} âą Publishes the received events to a WebSocket on port 4019, which NiFi subscribes to. Crypto Currency Data Feed
- 13. The Demo
- 14. âą Amount of effort to produce the demo was approximately 2 weeks, with the bulk of the time spent on the Apache NiFi processors. Source code for the Apache NiFi Processors (will be contributed back to the Apache project) can be found here: âą https://github.com/openconnectors/niïŹ-pulsar-bundle âą https://github.com/openconnectors/niïŹ-pulsar-client-services âą Infrastructure is Google Cloud Compute, managed by Kubernetes. Helm Charts can be found here: âą https://storage.googleapis.com/streamlio/charts âą Docker images are currently available in DockerHub here: âą https://hub.docker.com/r/streamlio/niïŹ/ âą https://hub.docker.com/r/streamlio/crypto-currency/ âą Phase 2 of the demo will add automated trading using Machine Learning automated by Apache Heron. Final Comments
- 16. Curious to Learn More? âą Apache Pulsar : http://pulsar.incubator.apache.org âą Apache DistributedLog : http://bookkeeper.apache.org/distributedlog âą Apache BookKeeper : http://bookkeeper.apache.org âą Follow Us @apache_pulsar @asfbookkeeper @distributedlog 16
- 17. Curious to learn more about Streamlio? âą Streamlio: https://streaml.io âą Sandbox Preview: https://streaml.io/docs/getting-started âą Learn slack channel: https://learn-streamlio.slack.com 17
- 18. Appendix
- 19. NiFi Flow
- 20. NiFi Flow Cont.