Serverless London 2019 FaaS composition using Kafka and CloudEvents
3 likes1,125 views
The document discusses Function as a Service (FaaS) and its integration with Kafka and CloudEvents, highlighting event-driven architectures and various composition styles such as orchestration and choreography. It emphasizes the growth of serverless computing and stream processing, detailing different processing patterns and the use of Kafka Streams for event handling. Additionally, it introduces CloudEvents as a specification for event data interoperability across services and platforms.
![22
Kafka Streams
● Data driven (dataflow)
● Partition aware
● Scale
● Resilient
● Delivery guarantees
● Replay events and time-travel
[Java, Scala]](https://image.slidesharecdn.com/serverlesslondon2019-faascompositionusingkafkaandcloud-events-191107100904/85/Serverless-London-2019-FaaS-composition-using-Kafka-and-CloudEvents-22-320.jpg)
![{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using a ‘unit-of-work’ pattern
{
digest.merge(digest)
}
Stream processor with
a session window to
join related events by
key (k, v[1-10])
Stream
processor
fan out is natural (data-driven)
OR
use a Stream processor flatMap()](https://image.slidesharecdn.com/serverlesslondon2019-faascompositionusingkafkaandcloud-events-191107100904/85/Serverless-London-2019-FaaS-composition-using-Kafka-and-CloudEvents-26-320.jpg)
Serverless London 2019 FaaS composition using Kafka and CloudEvents
- 1. 1 FaaS Composition using Kafka and CloudEvents Neil Avery, Office of the CTO, @avery_neil {...}
- 2. What is FaaS? ● ● ● ● ● ● ● DRIVEN BY EVENTS {...}
- 3. Origins of FaaS ● ● ● ● “It’s easy to run without managing infrastructure. It will take care of managing, scaling, monitoring and logging for you. Your applications run milliseconds after the event has been triggered and thousands of functions can work in parallel. You run the code only when needed.”
- 4. 44
- 5. 55
- 6. 66 Microservice/FaaS/Stream-Processor Single atomic unit does one thing like
- 7. 77 Composition <events don’t exist in isolation>
- 9. 99 Composition styles 1. Wiring 2. Chaining 3. Orchestration / Workflow (Step functions, Durable Fns, Zeebe, Apache Airflow) 4. Choreography
- 10. 1010 1. Wiring: Infrastructure events Triggered by S3 bucket, DynamoDB, SNS or CloudWatch
- 11. 1111 1. Wiring: REST Endpoint Http Request payload
- 12. 1212 2. Call chaining Send event from one Fn to other(s) Sync or Async(s) What could go wrong? < debug, logs, testing, portability>
- 13. 1313 3. Orchestration / Workflow Vendor: AWS Step functions, Azure Durable Fns OpenSource: Zeebe, Apache Airflow, CNCF Serverless workflow spec
- 14. 14
- 15. 15
- 16. 1616 3. Orchestration using a DataFlow
- 17. 1717
- 18. 1818 Rate of adoption ● Serverless is the top-growing extended cloud service for the second year in a row, with a 50 percent growth over 2018 24 to 36 percent adoption. ● Stream processing is tied for first, increasing from 20 to 30 percent adoption. Rightscale cloud report 2019
- 19. 1919 What is stream processing? Event stream processing, or ESP, is a set of technologies designed to assist the construction of event-driven information systems. ESP technologies include event visualization, event databases, event-driven middleware, and event processing languages, or complex event processing (CEP). SELECT DataStream Orders.TimeStamp, Orders.orderId, Orders.ticker,Orders.amount, Trade.amount FROM Orders JOIN Trades OVER (RANGE INTERVAL '1' SECOND FOLLOWING) ON Orders.orderId = Trades.orderId; https://en.wikipedia.org/wiki/Event_stream_processing
- 20. 20 Streaming processing as SQL Stream processor STREAM /user-reg FILTER SELECT users > 18 PROJECT SELECT user.name JOIN SELECT u.name, a.country from user-reg u JOIN address a WHERE u.id = a.id GROUP BY (TABLE) SELECT u.country, count(u.name) FROM user-reg u GROUP BY u.country WINDOW (TABLE) SELECT u.country, count(u.name) FROM user-reg u WINDOW TUMBLING (SIZE 1 MIN) group by u.country STREAM /address
- 21. 21 1. Stream processor runs ‘future’ on local-state of item-bids auction items 3. Rejoined removes item 4. Stream triggers FaaS complete item processor bid-history Stream table join 5. FaaS Notify all bidders bid-notifications 2. Rewrite ‘completed’ item status to retrigger join item status
- 22. 22 Kafka Streams ● Data driven (dataflow) ● Partition aware ● Scale ● Resilient ● Delivery guarantees ● Replay events and time-travel [Java, Scala]
- 23. 23 { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.filter( (key, payment) -> payment.isComplete()).to(TASK_COMPLETE); … Topology gateTopology = builder.build(); streams = new KafkaStreams(gateTopology, streamsConfig); streams.start(); }
- 24. 24 { StreamsBuilder builder = new StreamsBuilder(); KStream<String, Payment> incoming = builder.stream(TASK_TOPIC); incoming.flatMapValues(payment -> { ArrayList<String> fanTasks = new ArrayList<>(); for (int i = 0; i < payment.getAmount(); i++) { fanTasks.add("fan-" + i); } return fanTasks; } ).to(TASK_FANOUT); flatMapValues: Many V to the same K Or flatMap: Many K,V to one K,V pair
- 25. 25 builder.stream(TASK_FANOUT) .groupByKey(Grouped.with(Serdes.String(), Serdes.String())) .windowedBy(SessionWindows.with(Duration.ofMinutes(1))) .reduce(new Reducer<String>() { public String apply(String agg1, String newValue) { return agg1 + newValue; } }) .toStream() .map((key, value) -> new KeyValue<>(key.key() + "@" + key.window().start() + "->" + key.window().end(), value)) .to(TASK_REDUCE); or: groupBy(KeyValueMapper)
- 26. {...} {...} {...} { bid:100; sold:1000;} /bid-history { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } { stream-lib.TDigest(values[]) } Calculating percentiles using a ‘unit-of-work’ pattern { digest.merge(digest) } Stream processor with a session window to join related events by key (k, v[1-10]) Stream processor fan out is natural (data-driven) OR use a Stream processor flatMap()
- 27. 27 Stream processing ‘FaaS’ Patterns ● Filter (gate) ● Transform ● Fan-out (one to many) ● Fan-in (many to one - join) ● Branch/Choose (if ‘n’ then) ● Aggregate ● ...more...
- 28. 2828 A Stream processing DataFlow is both workflow and choreography (EDA, SEDA, Event-driven, streaming)
- 29. 2929 Choreography 1. Fire and forget / Async events 2. Producers emit state (not commands) 3. Consumers react to events 4. There is no master plan, no central brain LOOSE COUPLING
- 30. 30 Choreography
- 32. 32 An Event records the fact that something happened 32 A good was sold An invoice was issued A payment was made A new customer registered
- 34. 34 Evolvability user experience? how many users affected? has it happened before? new old supports data change, logic change, logic extension, schema evolution, loose coupling, add processors, A/B path
- 35. 35 is a distributed event streaming platform Publish & Subscribe to Events Store Events Process & Analyze Events
- 37. 37 payment.incoming.pre 2. Payment Inflight payment.incoming 1. {...} Fraudulent Check “FaaS Connector” a bridge between Kafka topic(s) and serverless function(s)
- 39. 39 FaaS Connector modes 1. Sync: ● fire and queue response ● preserve order 2. Async: ● fire and forget ● order agnostic https://www.confluent.io/hub → AWS Lambda Connector (preview) → Azure Functions Connector (preview) → GCP Functions Connector (preview) <confluent-platform> ../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties
- 40. 40 payment.incoming.pre 2. Payment Inflight payment.incoming 1. {...} Fraudulent Check Connector Bridge between Kaka topic and Fn invocation Producer Write back to Kafka?
- 42. 42 Bad stream? As we say in IT...
- 43. 4343
- 44. 44 Explain FaaS Connector modes 1. Sync: use case (ordered) - state change analytics: add then remove values; i.e. CDC - by topic partition 2. Async: use case (not ordered) - compute intensive analytics; what-if scenario analysis - machine learning training - ETL <confluent-platform> ../bin/connect-standalone ./etc/json-standalone.properties ./etc/AwsLambdaSinkConnector.properties
- 45. 45 Kafka Streams is a Java app…. It’s not very ‘Cloud native’ - without 45 Kubeless Cloud Foundry Knative GraalVm Quarkus
- 47. {faas} Central nervous system appappappapp Payments Department 2 {faas}appappappapp Department 3 Department 4 Choreography Choreography Orchestration Hierarchical topic naming
- 49. {faas} What is the message format James? appappappapp Payments Department 2 I’m so confused!
- 51. 51
- 52. 52 ● CNCF ● Version 1.0 October 2019 ● Specification (an Event envelope) ● Transport Bindings (Kafka, AMQP, HTTP others) ● SDKs: Java, .NET, Go-lang, Javascript etc ● Useful for exposing to external apps (Events as APIs) CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems.
- 53. 5353 CloudEvent semantics ● Self describing / self contained / stand alone ● Suppressed / non-intruisive ● Language, transport, data format independent ● Propagation ● Extensions ● Batteries included
- 54. 54 { "specversion" : "0.4-wip", "type" : "com.github.pull.create", "source" : "https://github.com/cloudevents/spec/pull", "subject" : "123", "id" : "A234-1234-1234", "time" : "2018-04-05T17:31:00Z", "comexampleextension1" : "value", "comexampleextension2" : { "othervalue": 5 }, "datacontenttype" : "text/xml", "data" : "<much wow="xml"/>" }
- 55. 55 CloudEvents Producer Consumer SerDes Kafka Streams and KSQL SerDes Kafka Connect Convertor Cloud Function Connectors AWS, GCP, Azure and the Kafka Ecosystem
- 56. 56 KStream<String, CloudEvent<AttributesImpl, Payment>> payments = builder.stream(paymentTopic, Consumed.with(StringSerde, new CeBinarySerDes<>(Payment.class))); payments.filter((k,v) -> { return v.getData().get().getOrigin().contains("UK"); }) CloudEvents with Kafka Streams
- 57. {faas} Central nervous system appappappapp Payments Department 2 {faas}appappappapp Department 3 Department 4 Choreography Orchestration Choreography Orchestration Zeebe Hierarchical topic naming Kafka Streams Quarkus GraalVM KNative CloudEvents Confluent Cloud
- 58. 5858 Where to go from here for more details on event-driven architectures with Kafka @avery_neil
- 59. 59 Questions? @avery_neil “Journey to event driven” blog 1. Event-first thinking 2. Programming models 3. Serverless 4. Pillars of event-streaming ms’s https://bit.ly/2tFfU84 or @avery_neil twitter profile
- 60. 60 @avery_neil
- 61. 61 Cloud Native Kafka Streams (wow!) 61 1. StreamsTopology.java → docker-compose up --build. → topology.q-native-container 2. Create a Knative service descriptor topology-java-quarkus.yaml 3. Run locally: ./mvnw compile quarkus:dev 4. Publish docker push {username}/topology-java-quarkus 5. Deploy kubectl apply --filename service.yaml 6. Observe: kubectl get ksvc topology-java-quarkus NAME URL : topology-java-quarkus http://topology-java-quarkus.default.1.2.3.4.xip.io https://quarkus.io/guides/kafka-streams-guide https://knative.dev/community/samples/serving/helloworld-java-quarkus/