Pulsar connector on flink 1.14
1 like149 views
This document discusses the Pulsar connector for Apache Flink 1.14. It provides an overview of StreamNative, which offers both stream storage with Apache Pulsar and stream processing with Flink. It then covers the timeline of contributions to the Pulsar connector for Flink and how it has evolved. Finally, it describes the design of the new Pulsar source connector for Flink that uses the FLIP-27 source interface, including how it handles Pulsar subscription modes and implements split enumeration, reading, and processing in a way that supports both batch and streaming workloads.
Pulsar connector on flink 1.14
- 1. streamnative.io Pulsar Connector on Flink 1.14 The Path To Unified Batch and Streaming Yufan Sheng
- 2. streamnative.io This document and all information and content contained herein are protected by China copyright and other intellectual property laws, and are the confidential information of StreamNative, Inc. No part of this document may be reproduced, distributed or displayed in any form or by any means, or used to make any derivative work (such as a translation or adaptation), without prior written permission from StreamNative, Inc. Copyright © 2021 StreamNative, Inc. All rights reserved.
- 3. streamnative.io Lecturer Introduction Yufan Sheng is a software engineer at StreamNative where he devotes in flink or other streaming platform integration with Apache Pulsar. Before that he was a senior software engineer at Tencent Cloud. He was the core committer to the Barad project (Tencent Cloud Monitor Analysis Platform) and the leading engineer for Barad development at Tencent. @syhily @syhily yufan@streamnative.io
- 4. streamnative.io Introduction to StreamNative Developed by the original creators of Apache Pulsar and Apache Bookkeeper, StreamNative provides a real-time streaming platform built for Kubernetes. StreamNative Cloud offers a turnkey platform to help you scale mission-critical applications, on enterprise-grade security and compliance.
- 5. streamnative.io StreamNative is building a complete streaming solution With Pulsar and Flink, StreamNative offers both stream storage and stream compute for a complete streaming solution.
- 6. streamnative.io Timeline of Pulsar connector contribution 2018/4 A simple producer & table sink FLINK-9168 2019/11 Pulsar catalog FLINK-15089 2021/1 Pulsar source on legacy SourceFunction FLINK-20727 2018/6 Pulsar producer FLINK-9641 2020/12 Announced on Flink Forward 2020 Pulsar-Flink 2.7.0 2021/8 Pulsar source on FLIP-27 FLINK-20731
- 7. streamnative.io Subhead streamnative.io Unified APIs for Batch and Stream Processing on Flink
- 8. streamnative.io Batch Processing is a special case of Stream Processing A batch is just a bounded subset of stream(bounded stream).
- 9. streamnative.io Batch vs Stream execution Stream Processor ● Bounded execution ● Unbounded execution Batch Processor ● Bounded execution ● Unbounded execution
- 10. streamnative.io Source boundary determines Stream type That is about 60% of the truth… Source Job All source bounded => Job is bounded One source unbounded => Job is unbounded
- 11. streamnative.io The remaining 40% of the truth … never seen this in Batch Processing, though. The (Event-time) Watermark *from the excellent Streaming 101 by Tyler Akidau: https://www.oreilly.com/ideas/the-world-beyond-batch-streaming101)
- 12. streamnative.io Inefficiencies in Job Semantics ● Watermarks, lateness, late firing, early firing, completeness vs. latency ● Incremental results vs. definitive, final results ● Processing time As a user, you don‘t want to worry about these when executing a bounded job on the DataStream API!
- 13. streamnative.io The remaining 40% of the truth Data is incomplete Latency SLAs Completeness and Latency is a tradeoff ETL pipelines, standing queries, anomaly detection, ML evaluation Continuous Streaming Data is as complete as it gets within the job No Low Latency SLAs ad-hoc queries, data exploration, ML training Batch Processing
- 14. streamnative.io Flink legacy (before 1.9) API stack Streaming Runtime DataStream API DataSet API Batch Libraries Table API / SQL Streaming Libraries ● DataSet API: batch processing ● DataStream API: stream processing ● Table API/SQL: relational (batch and stream)
- 15. streamnative.io Legacy Source Interfaces Streaming: SourceFunction Source Thread push() Batch: InputFormat Enumerate splits read splits pull() pull() JobManager
- 16. streamnative.io Batch InputFormat Processing TaskManager TaskManager TaskManager JobManager (3) process split (1) request split (2) send split InputFormat createInputSplits(): splits assignInputSplit() openSplit(split) nextRecord(): T closeCurrentSplit()
- 17. streamnative.io TaskManager TaskManager TaskManager JobManager (3) process split (1) request split (2) send split ● Splits are assigned to TaskManagers by the JobManager, which runs a copy of the InputFormat => Flink knows about splits and can be clever about scheduling, be reactive ● Splits can be processed in arbitrary order ● Split processing pulls records from the InputFormat ● InputFormat knows nothing about watermarks, timestamps, checkpointing => bad for streaming
- 18. streamnative.io Stream SourceFunction Processing SourceFunction run(OutputContext) close() (1) do your thing TaskManager TaskManager TaskManager ● Source have a run-loop that they manage completely on their own ● Sources have flexibility and can efficiently work with the source system: batch accesses, dealing with multiple topics from one consumer, threading model, etc… ● Flink does not know what’s going on inside and can’t be clever about it ● Sources have to implement their own per-partition watermarking, idleness tracking, what have you
- 19. streamnative.io A New (unified) Source Interface (FLIP-27) Source createSplitEnumerator() createSplitReader() SplitEnumerator discoverNewSplits() nextSplit() isDone() snapshotState() SplitReader addSplit() hasAvailable(): Future emitNext(Context): Status snapshotState() ● This must support both batch and streaming use cases, allow Flink to be clever, be able to deal with event-time, watermarks, source idiosyncrasies, and enable snapshotting. ● This should enable new features: generic idleness detection, event-time alignment.* * FLINK-10886: Event-time alignment for sources
- 20. streamnative.io Execution Style TaskManager TaskManager TaskManager JobManager (3) process split (1) request split (2) send split
- 23. streamnative.io Pulsar Subscription Modes Different subscription modes have different semantics: Exclusive/Failover - guaranteed order, single active consumer Shared - multiple active consumers, no order Key_Shared - multiple active consumers, order for given key Subscription D Consumer D-1 Consumer D-2 Key-Shared < K 1, V 10 > < K 1, V 11 > < K 1, V 12 > < K 2 ,V 2 0 > < K 2 ,V 2 1> < K 2 ,V 2 2 > Subscription C Consumer C-1 Consumer C-2 Shared < K 1, V 10 > < K 2 ,V 2 1> < K 1, V 12 > < K 2 ,V 2 0 > < K 1, V 11 > < K 2 ,V 2 2 > Subscription A Consumer A Exclusive Subscription B Consumer B-1 Consumer B-2 In case of failure in Consumer B-1 Failover Pulsar Partition is also a sub set of topics which has only one partition and could be consumed directly. Producer 1 Producer 2 Pulsar Topic m0 m1 m2 m3 m4
- 24. streamnative.io Pulsar Split design PulsarPartitionSplit TopicPartition partition StartCursor startCursor StopCursor stopCursor MessageId latestConsumedId TxnID uncommittedTransactionId TopicPartition String topic int partitionId TopicRange range wrap into 1 to 1 Partitioned Topic Non-Partitioned Topic 1 to n 1 to 1 For Exclusive, Failover and Shared Failover Consumer B-1 Consumer B-0 Subscription B m1 m2 m3 m4 m0 In case of failure in Consumer B-0 Consumer A-1 Consumer A-0 Subscription A m1 m2 m3 m4 m0 Exclusive X Consumer C-1 Consumer C-2 Consumer C-3 Subscription C Shared < k 1 , v 0 > < k 1 , v 4 > < k 3 , v 2 > <k 2, v1 > < k 2 , v 3 >
- 25. streamnative.io Pulsar Split design For Key_Shared TopicPartition String topic int partitionId Partitioned Topic Non-Partitioned Topic 1 to n 1 to 1 RangeGenerator TopicPartition String topic int partitionId TopicRange range 1 to n … Shared and Key_Shared can share the same subscription for all consumers, why we create the split on partition? Consumer D-1 Consumer D-2 Consumer D-3 Subscription D < k 2 , v 1 > < k 2 , v 3 > <k3 ,v2 > < k 1 , v 0 > < k 1 , v 4 > Key-Shared
- 26. streamnative.io Pulsar Split design Create the split on partition Consumer.seek() can only be executed on single topic with one partition or no partition.
- 27. streamnative.io Pulsar Enumerator design Pull splits Enumerator TopicListSubscriber TopicPatternSubscriber Continuous unbound (Stream) would cycle query Bounded (Batch) would query only once
- 28. streamnative.io Pulsar Enumerator design Assign split on Exclusive, Failover and Key_Shared subscription Enumerator Reader 1 Reader 2 Reader 3 Split 3 Split 4 Split 7 Split 2 Split 9 Split 6 Split 1 Split 5 Split 8 Consumer D-1 Consumer D-2 Consumer D-3 Subscription D < k 2 , v 1 > < k 2 , v 3 > <k3 ,v2 > < k 1 , v 0 > < k 1 , v 4 > Key-Shared Failover Consumer B-1 Consumer B-0 Subscription B m1 m2 m3 m4 m0 In case of failure in Consumer B-0 Consumer A-1 Consumer A-0 Subscription A m1 m2 m3 m4 m0 Exclusive X
- 29. streamnative.io Pulsar Enumerator design Assign split on Shared subscription Enumerator Reader 1 Reader 2 Reader 3 Split 1 Split 2 Split 3 Split 1 Split 3 Split 2 Split 1 Split 2 Split 3 Why don’t share splits for Key_Shared subscription. Consumer C-1 Consumer C-2 Consumer C-3 Subscription C Shared < k 1 , v 0 > < k 1 , v 4 > < k 3 , v 2 > <k 2, v1 > < k 2 , v 3 >
- 30. streamnative.io Pulsar Reader design Ordered message consuming (Failover, Exclusive) Ordered Reader Ordered Split Reader SortedMap<Long, Map<TopicPartition, MessageId>> cursorsToCommit ConcurrentMap<TopicPartition, MessageId> cursorsOfFinishedSplits ScheduledExecutorService cursorScheduler Used cumulative message acknowledge, we only memorize last consumed message id. Acknowledge message with id when flink finished the checkpointing. Failover Consumer B-1 Consumer B-0 Subscription B m1 m2 m3 m4 m0 In case of failure in Consumer B-0 Consumer A-1 Consumer A-0 Subscription A m1 m2 m3 m4 m0 Exclusive X
- 31. streamnative.io Pulsar Reader design Unordered message consuming (Shared, Key_Shared) Unordered Reader Unordered Split Reader TransactionCoordinatorClient coordinatorClient SortedMap<Long, List<TxnID>> transactionsToCommit List<TxnID> transactionsOfFinishedSplits Acknowledge message in a transaction, we only hold one transaction at a time. Commit the transaction when flink finished the checkpointing. Consumer D-1 Consumer D-2 Consumer D-3 Subscription D < k 2 , v 1 > < k 2 , v 3 > <k3 ,v2 > < k 1 , v 0 > < k 1 , v 4 > Key-Shared Consumer C-1 Consumer C-2 Consumer C-3 Subscription C Shared < k 1 , v 0 > < k 1 , v 4 > < k 3 , v 2 > <k 2, v1 > < k 2 , v 3 >
- 32. streamnative.io Flink type system vs Pulsar Schema Pulsar ● Schema: Client side, used for serialize and deserialize message. ● SchemaInfo: Server and Client side, used for compatible validation and schema evolution. ● SchemaDefinition: Client side, used for creating a Pulsar Schema. Flink ● (De)serializationSchema: Operator side, used for deserialize source element and provide a TypeInformation. ● TypeInfomation: Shared among TaskManagers, used for message serialization and transportation. The flink’s default type systems, would fallback to Kyro when no types provided. ● TypeSerializer: Used for message serialize and deserialize.
- 33. streamnative.io Pulsar Schema on Flink Schema.BYTES Reader 1 DeserializationSchema<T> Downstr eam Schema.BYTES Reader 1 Schema<T> TypeInformation<T> Downstr eam
- 34. streamnative.io Pulsar Schema evolution on Flink Schema<T> Reader 1 TypeInformation<T> Downstr eam Make Pulsar Schema serializable. Auto mapping Schema to Flink TypeInformation.
- 35. streamnative.io Checkpoint on Pulsar Source Job Coordinator Enumerator Ordered Reader Ordered Split Reader Unordered Reader Unordered Split Reader 1. Triggered a checkpoint 2. Snapshot the split assign status. 1. Triggered a checkpoint 2. Snapshot the consuming position. 3. Save position with checkpoint id. 4. Notify checkpoint complete 5. Acknowledge message id to pulsar 1. Triggered a checkpoint 2. Start a new transaction. 3. Save transaction id with checkpoint id. 4. Notify checkpoint com plete 5. Com m it transaction
- 36. streamnative.io Pulsar Source configuration We have exposed all the configuration options in: • PulsarSourceOptions: Pulsar Source and Pulsar Consumer config option. • PulsarOptions: Options for PulsarAdmin and PulsarClient builder. These options are define by flink’s ConfigOption, with strong type support.
- 37. streamnative.io Overview of PulsarSourceBuilder PulsarSource<String> source = PulsarSource .builder() .setServiceUrl(PULSAR_BROKER_URL) .setAdminUrl(PULSAR_BROKER_HTTP_URL) .setSubscriptionName("flink-source-1") .setTopics(Arrays.asList(TOPIC1, TOPIC2)) .setDeserializationSchema(PulsarDeserializationSchema.flinkSchema(new SimpleStringSchema())) .setUnbounded(StopCursor.atEventTime(System.currentTimeMillis())) .build(); A simple sample on consuming string messages from Pulsar
- 38. streamnative.io You should provide at least these information. setServiceUrl setAdminUrl setSubscriptionName setTopics/setTopicPattern setDeserializationSchema Overview of PulsarSourceBuilder
- 39. streamnative.io Create PulsarDeserializationSchema You can create a PulsarDeserializationSchema in three ways: 1. Using a Pulsar Schema, we would auto create it’s related flink TypeInformation. 2. Using a Flink DeserializationSchema, we would use pure flink mechanism. 3. Using a Flink TypeInformation, it’s not recommend for it doesn’t compatible with others above.
- 40. streamnative.io Provide StartCursor and StopCursor StartCursor could be created in four ways. 1. earliest(): From the earliest available message in the topic. 2. latest(): From the latest available message in the topic. 3. fromMessageId(MessageId messageId, boolean inclusive): From a specified message id. 4. fromMessageTime(long timestamp): From a specified message time. Caution: 1. If the provided message id/time is bigger than the available message, we would start consuming from the latest(). 2. If the provided message id/time doesn’t existed (deleted), we would find the next available message, it may fallback to the latest() finally.
- 41. streamnative.io Provide StartCursor and StopCursor StopCursor could be created in five ways. 1. never(): We never stop, the source is infinite. 2. latest(): Query the latest message and stop after this message id. 3. atMessageId(MessageId messageId): Stop at a specified message id (exclude). 4. afterMessageId(MessageId messageId): Stop after a specified message id (include). 5. atEventTime(long timestamp): Stop at a specified message time (exclude). Caution: latest() is used for test and batch data consuming. You should fully understand its internal logic before using.
- 42. streamnative.io Follow us StreamNative Apache Pulsar Pulsar Bot
- 43. streamnative.io