[DoKDayNA2022] - Architecting Your First Event Driven Serverless Streaming Applications on K8
1 like696 views
Tim Spann, a developer advocate with over 15 years of experience, presents on architecting an event-driven serverless streaming application using Apache Pulsar at KubeCon 2022. The presentation covers key concepts of Pulsar, its advantages for cloud-native architectures, and how to use Pulsar functions for processing messages across various programming languages. The talk also emphasizes the importance of messaging versus streaming paradigms and showcases tools and resources for deploying Pulsar in Kubernetes environments.
![Tim Spann
“Serverless
Streaming”
from pulsar import Function
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
import json
class Chat(Function):
def __init__(self):
pass
def process(self, input, context):
fields = json.loads(input)
sid = SentimentIntensityAnalyzer()
ss = sid.polarity_scores(fields["comment"])
row = { }
row['id'] = str(msg_id)
if ss['compound'] < 0.00:
row['sentiment'] = 'Negative'
else:
row['sentiment'] = 'Positive'
row['comment'] = str(fields["comment"])
json_string = json.dumps(row)
return json_string
Entire Function
Pulsar Python
NLP Function
https://github.com/tspannhw/pulsar-pychat-function](https://image.slidesharecdn.com/dokdayna2022-architectingyourfirsteventdrivenserverlessstreamingapplicationsonk8-221028122527-9da3a03d/85/DoKDayNA2022-Architecting-Your-First-Event-Driven-Serverless-Streaming-Applications-on-K8-10-320.jpg)
[DoKDayNA2022] - Architecting Your First Event Driven Serverless Streaming Applications on K8
- 1. Tim Spann “Serverless Streaming” “Architecting Your First Event Driven Serverless Streaming Application on K8”
- 2. Tim Spann “Serverless Streaming” Tim Spann DoK Day North America 2022 @ KubeCon “Serverless Streaming” Timothy Spann Developer Advocate FLiP(N) Stack = Flink, Pulsar and NiFi Stack Streaming Systems & Data Architecture Expert Experience: 15+ years of experience with streaming technologies including Pulsar, Flink, Spark, NiFi, Kafka, Big Data, Cloud, MXNet, IoT and more. Today, he helps to grow the Pulsar community sharing rich technical knowledge and experience at both global conferences and through individual conversations.
- 3. Tim Spann “Serverless Streaming” Tim Spann DoK Day North America 2022 @ KubeCon “Serverless Streaming” FLiP Stack Weekly This week in Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark, Elasticsearch and open source friends. https://bit.ly/32dAJft
- 4. Tim Spann “Serverless Streaming” Tim Spann “Serverless Streaming” Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform. DoK Day North America 2022 @ KubeCon
- 5. Tim Spann “Serverless Streaming” Apache Pulsar adoption is being driven by organizations seeking cloud-native architectures and new uses cases. Microservices Apache Pulsar - Built for Containers / Modern Cloud Cloud Native Hybrid & Multi-Cloud Containers DoK Day North America 2022 @ KubeCon
- 6. Tim Spann “Serverless Streaming” Key Pulsar Concepts: Messaging vs Streaming Message Queueing - Queueing systems are ideal for work queues that do not require tasks to be performed in a particular order. Streaming - Streaming works best in situations where the order of messages is important. DoK Day North America 2022 @ KubeCon
- 7. Tim Spann “Serverless Streaming” Pulsar Cluster CLUSTER Global ZooKeeper Configuration Store ZK3 ZK2 ZK1 Local ZooKeeper Quorum Bookie 0 Bookie 1 Bookie 2 BookKeeper Bookie Ensemble Pulsar Broker 0 Pulsar Broker 1 Pulsar Broker 2 DoK Day North America 2022 @ KubeCon
- 8. Tim Spann “Serverless Streaming” ● Consume messages from one or more Pulsar topics. ● Apply user-supplied processing logic to each message. ● Publish the results of the computation to another topic. ● Support multiple programming languages (Java, Python, Go) ● Can leverage 3rd-party libraries Pulsar Functions DoK Day North America 2022 @ KubeCon
- 9. Tim Spann “Serverless Streaming” import java.util.function.Function; public class MyFunction implements Function<String, String> { public String apply(String input) { return doBusinessLogic(input); } } Entire Function Why Pulsar Functions?
- 10. Tim Spann “Serverless Streaming” from pulsar import Function from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer import json class Chat(Function): def __init__(self): pass def process(self, input, context): fields = json.loads(input) sid = SentimentIntensityAnalyzer() ss = sid.polarity_scores(fields["comment"]) row = { } row['id'] = str(msg_id) if ss['compound'] < 0.00: row['sentiment'] = 'Negative' else: row['sentiment'] = 'Positive' row['comment'] = str(fields["comment"]) json_string = json.dumps(row) return json_string Entire Function Pulsar Python NLP Function https://github.com/tspannhw/pulsar-pychat-function
- 11. Tim Spann “Serverless Streaming” Function Mesh Pulsar Functions, along with Pulsar IO/Connectors, provide a powerful API for ingesting, transforming, and outputting data. Function Mesh, another StreamNative project, makes it easier for developers to create entire applications built from sources, functions, and sinks all through a declarative API.
- 13. Tim Spann “Serverless Streaming” Function Execution DoK Day North America 2022 @ KubeCon
- 14. Tim Spann “Serverless Streaming” Apache Pulsar Kafka K8 https://docs.streamnative.io/platform/v1.3.0/quickstart DoK Day North America 2022 @ KubeCon
- 15. Tim Spann “Serverless Streaming” Webinar Series: Building Microservices with Pulsar Watch now Read more Learn how Pulsar Functions, can be leveraged to build a message bus for event-driven microservices DoK Day North America 2022 @ KubeCon
- 16. Tim Spann “Serverless Streaming” Apache Pulsar Resources ● https://pulsar.apache.org/docs/en/kubernetes-helm/ ● https://pulsar.apache.org/charts ● https://streamnative.io/streamnativecloud/ ● https://medium.com/@_oleksii_/how-to-deploy-apache-puls ar-cluster-in-kubernetes-808ecdfd87 Academy.StreamNative.io
- 17. Tim Spann “Serverless Streaming” Thank you! DoK Day North America 2022 @ KubeCon Tim Spann “Serverless Streaming”