Pulsar - flexible pub-sub for internet scale
4 likes1,230 views
Apache Pulsar is a flexible, multi-tenant pub-sub messaging system designed for large-scale internet applications, offering features like high throughput, low latency, and geo-replication. It evolved from initial development at Yahoo in 2012 to becoming an Apache project, addressing scalability and operational challenges faced by existing systems. The architecture includes separate layers for brokers and storage, and it supports multiple client libraries and functions for processing messages.
![GO CLIENT
• go get -u github.com/apache/pulsar/pulsar-client-go/pulsar
client, err := pulsar.NewClient(pulsar.ClientOptions{
URL: "pulsar://localhost:6650"
})
producer, err := client.CreateProducer(pulsar.ProducerOptions{
Topic: "my-topic",
})
for i := 0; i < 10; i++ {
err := producer.Send(context.Background(), pulsar.ProducerMessage{
Payload: []byte(fmt.Sprintf("hello-%d", i)),
})
}
• Based on C++ client library — Pure Go client is being worked on
20](https://image.slidesharecdn.com/pulsar-flexiblepub-subforinternetscale-181002061232/85/Pulsar-flexible-pub-sub-for-internet-scale-20-320.jpg)
Pulsar - flexible pub-sub for internet scale
- 1. APACHE PULSAR Flexible Pub-Sub system for Internet scale http://pulsar.apache.org Content on this presentation is licensed under a Creative Commons Attribution 4.0 International license
- 2. Pulsar graduates asTLP project today!
- 3. WHO AM I? • Matteo Merli • Apache Pulsar PMC Chair • Member of Apache BookKeeper PMC • Co-Founder of Streamlio • Worked on Pulsar since its beginning atYahoo
- 4. WHAT IS APACHE PULSAR? “Pub-Sub messaging backed by durable log storage”
- 5. WHAT IS APACHE PULSAR? 5 Multi-tenancy A single cluster can support many tenants and use cases Ordering Guaranteed ordering Durability Data replicated and synced to disk Delivery Guarantees At least once, at most once and effectively once Highly scalable Can support millions of topics Unified messaging model Support both Topic & Queue semantic in a single model Geo-replication Out of box support for geographically distributed applications High throughput Can reach 1.8 M messages/s in a single partition Low Latency Low publish latency of 5ms at 99pct
- 6. WHY BUILD A NEW SYSTEM? • No existing solution to satisfy requirements • Multi tenant — 1M topics — Low latency — Durability — Geo replication • Other systems don’t scale well with many topics: • Storage model based on individual directory per topic partition • Durability kills the performance • Ability to manage large backlogs — Read old data without impacting writers • Many other choking points: getting stats, access to metadata, flow-control • Operations are not very convenient — replacing servers, expanding clusters, etc… 6
- 7. STATE OFTHE PROJECT • Project started atYahoo around 2012 and went through various iterations • Open-Sourced in September 2016 • Entered Apache Incubator in June 2017 • Graduated asTLP on September 2018 • 2249 Commits — 22Yahoo releases — 9 Apache releases • 59 Contributors 7
- 8. ARCHITECTURALVIEW Separate layers between brokers bookies • Broker and bookies can be added independently • Traffic can be shifted very quickly across brokers • New bookies will ramp up on traffic quickly
- 9. APACHE BOOKKEEPER Replicated log storage • Low-latency durable writes • Simple repeatable read consistency • Highly available • Store many logs per node • I/O Isolation
- 10. SEGMENT CENTRIC STORAGE • In addition to partitioning, messages are stored in segments (based on time and size) • Segments are independent from each others and spread across all storage nodes
- 12. DATA PATH 1 — Publisher sends message to broker
- 13. DATA PATH 2 — Broker writes in parallel to N replicas
- 14. DATA PATH 3 — Wait for a quorum of acks from bookies
- 15. DATA PATH 4 — Send ack to producer — Dispatch to consumer
- 16. BOOKKEEPER INTERNAL Storage optimized for sequential & immutable data • IO isolation between write and read operations • Slow consumers won’t impact latency • Very effective IO patterns: • Journal — append only and no reads • Storage device — bulk write and sequential reads • Number of files is independent from number of topics
- 17. MESSAGING MODEL
- 18. PULSAR CLIENT LIBRARY • Java — C++ — Python — Go — WebSocket APIs • Partitioned topics • Apache Kafka compatibility wrapper API • Transparent batching and compression • TLS encryption and authentication • End-to-end encryption 18
- 19. PYTHON CLIENT import pulsar client = pulsar.Client('pulsar://localhost:6650') producer = client.create_producer('my-topic') for i in range(10): producer.send(('Hello-%d' % i).encode('utf-8')) client.close() 19 • pip install pulsar-client
- 20. GO CLIENT • go get -u github.com/apache/pulsar/pulsar-client-go/pulsar client, err := pulsar.NewClient(pulsar.ClientOptions{ URL: "pulsar://localhost:6650" }) producer, err := client.CreateProducer(pulsar.ProducerOptions{ Topic: "my-topic", }) for i := 0; i < 10; i++ { err := producer.Send(context.Background(), pulsar.ProducerMessage{ Payload: []byte(fmt.Sprintf("hello-%d", i)), }) } • Based on C++ client library — Pure Go client is being worked on 20
- 21. MULTI-TENANCY • Authentication / Authorization / Namespaces / Admin APIs • I/O Isolations between writes and reads • Provided by BookKeeper - Ensure readers draining backlog won’t affect publishers • Soft isolation • Storage quotas — flow-control — back-pressure — rate limiting • Hardware isolation • Constrain some tenants on a subset of brokers or bookies 21
- 22. GEO REPLICATION Topic (T1) Topic (T1) Topic (T1) Subscription (S1) Subscription (S1) Producer (P1) Consumer (C1) Producer (P3) Producer (P2) Consumer (C2) Data Center A Data Center B Data Center C • Scalable asynchronous replication • Integrated in the broker message flow • Simple configuration to add/remove regions
- 23. SCHEMA REGISTRY • Store information on the data structure — Stored in BookKeeper • Enforce data types on topic • Allow for compatible schema evolutions 23
- 24. TYPE-SAFE CLIENT API Producer<MyClass> producer = client .newProducer(Schema.JSON(MyClass.class)) .topic("my-topic") .create(); producer.send(new MyClass(1, 2)); 24 Consumer<MyClass> consumer = client .newConsumer(Schema.JSON(MyClass.class)) .topic("my-topic") .subscriptionName("my-subscription") .subscribe(); Message<MyClass> msg = consumer.receive(); • Integrated schema in API • End-to-end type safety — Enforced in Pulsar broker
- 25. PULSAR FUNCTIONS Managed lightweight compute framework
- 26. PULSAR FUNCTIONS / 1 • Simple compute against a consumed message • Managed or manual deployment • A function gets messages from 1 or more topics • An instance of the function is invoked to process the event • The output of the function is published on 1 or more topics 26
- 27. PULSAR FUNCTIONS / 2 • Super simple to use — no SDK • Python example: def process(input): return input + '!' • Supports Java & Python — Go will come next 27
- 28. PULSAR FUNCTIONS / 3 • Good use cases for functions: • ETL • Data enrichment • Data filtering • Routing 28
- 29. PULSAR FUNCTIONS / 4 • Deployment modes: • Local run — Manually run a function, useful for dev mode • Managed — Worker service is running instances of functions 29
- 30. PULSAR IO Connector framework based on Pulsar Functions
- 31. PULSAR IO • Source — Ingest data into a Pulsar topic • Sink — Reads data from topic and dump into external sink • Pulsar provides a set of built-in connectors • Users can submit customized connectors 31
- 32. TIERED STORAGE Unlimited topic storage capacity Achieves the true “stream-storage”: keep the raw data forever in stream form
- 33. TIERED STORAGE • Leverage cloud storage services to offload cold data — Completely transparent to clients • Extremely cost effective — Backends (S3) (Coming GCS, HDFS) • Example: Retain all data for 1 month — Offload all messages older than 1 day to S3 33
- 34. PULSAR SQL • Coming very soon in Pulsar 2.2 • Interactive SQL queries over data stored in Pulsar • Query old and real-time data 34
- 35. PULSAR SQL / 2 • Based on Presto by Facebook — https://prestodb.io/ • Presto is a distributed query execution engine • Fetches the data from multiple sources (HDFS, S3, MySQL, …) • Full SQL compatibility 35
- 36. PULSAR SQL / 3 • Pulsar connector for Presto • Read data directly from BookKeeper — bypass Pulsar Broker • Many-to-many data reads • Data is split even on a single partition — multiple workers can read data in parallel from single Pulsar partition • Time based indexing — Use “publishTime” in predicates to reduce data being read from disk 36
- 38. BENCHMARK FRAMEWORK • Designed to measure performance of distributed messaging systems • Supports various “drivers” (Kafka, Pulsar, RocketMQ, RabbitMQ) • Automated deployment in EC2 • Configure workloads through aYAML file 38
- 39. DISTRIBUTED EXECUTION Coordinator will take the workload definition and propagate to multiple workers — Collects and reports stats
- 41. Latency at fixed throughput 50K msg/s 1Topic 1 Partition 1KB payload
- 42. Latency at fixed throughput — (including Kafka-sync) 50K msg/s 1Topic 1 Partition 1KB payload
- 43. Latency at fixed throughput — 99pct 50K msg/s 1Topic 1 Partition 1KB payload
- 44. Q & A