Data Con LA 2022 - Event Sourcing with Apache Pulsar and Apache Quarkus
0 likes44 views
This document discusses event sourcing with Apache Pulsar and Quarkus. It provides background on event sourcing and how it supports microservices architecture. Apache Pulsar is presented as a cloud-native event streaming platform that is well-suited for event sourcing due to its guaranteed message delivery, scalability, and other features. Quarkus is discussed as a Kubernetes-native Java framework that is optimized for container-based architectures. An example application architecture is shown that uses Pulsar for event storage and Quarkus microservices that generate views of the event data.
![streamnative.io
[Webinar]
Building Microservices
Watch Now Learn More
[Blog post]
Event-Driven Microservices](https://image.slidesharecdn.com/session6davidkjerrumgaardemergingtecheventsourcingwithapachepulsarandapachequarkus-220830164910-4950a383/85/Data-Con-LA-2022-Event-Sourcing-with-Apache-Pulsar-and-Apache-Quarkus-31-320.jpg)
Data Con LA 2022 - Event Sourcing with Apache Pulsar and Apache Quarkus
- 1. Event Sourcing with Apache Pulsar & Quarkus David Kjerrumgaard | Developer Advocate
- 2. streamnative.io ● Apache Pulsar Committer | Author of Pulsar In Action ● Former Principal Software Engineer on Splunk’s messaging team that is responsible for Splunk’s internal Pulsar-as-a-Service platform. ● Former Director of Solution Architecture at Streamlio. David Kjerrumgaard Developer Advocate
- 3. streamnative.io ● Author of Pulsar In Action. ● Co-Author of Practical Hive David Kjerrumgaard Author https://streamnative.io/download/manning-ebook-apache-pulsar-in-action
- 5. streamnative.io • DevOps automates the process of software delivery & infrastructure changes. • ContinuousDeliveryenables applications to released quickly and reliably with less risk. • Containersprovide light-weight virtualization by dynamically dividing a single server into one or more isolated containers. • Microservices is an architectural approach to building an application as a collection of small independent services that communicate with one another. * Image src: https://pivotal.io/cloud-native
- 6. streamnative.io • Componentization via Services • Organizedaround Business Capabilities • Products notProjects • SmartEndpoints & Dumb Pipes • Decentralized Governance • DecentralizedDataManagement • Infrastructure Automation • DesignforFailure • Evolutionary Design Microservice Characteristics
- 7. streamnative.io Decentralized Data Management ● At the most abstract level, the conceptual model of the world differs between microservices. ● Monolithic applications prefer a single logical database for persistent data. ● Microservices prefer letting each service manage its own database, either different instances of the same database technology, or entirely different database systems - an approach called Polyglot Persistence.
- 8. Polyglot Persistence * Image src: https://martinfowler.com
- 9. streamnative.io Design for Failure ● Since microservices can fail at any time, it's important to be able to automatically restore service. ● For partial failures scenarios, this requires processing all the events that occurred while the service was offline in order to update the internal database state. ● For complete failure events, the internal database must be reconstructed from scratch.
- 10. streamnative.io • Event Sourcing ensures that all changes to application state are stored as a sequence of events in a log. • We can use the event log to reconstruct past states, and as a foundation adjust the state to cope with retroactive changes. • Supports Polyglot Persistence by allowing each service to build it own database based on the events • In the event of failure, the service state can be rebuilt from the events in the event log. Event Sourcing
- 11. streamnative.io State Based vs. Event Based Data Models State-Based: ● Record the position/state at any point in time using a database and synchronous calls. ● Supports full CRUD semantics. Reads/Writes on the same data source.
- 12. streamnative.io State Based Data Model
- 13. streamnative.io State Based vs. Event Based Data Models State-Based: ● Record the position/state at any point in time using a database and synchronous calls. ● Supports full CRUD semantics. Reads/Writes on the same data source. Event-Based: ● Records all events in an immutable sequence. ● Each event represents a transition from one state to another. ● Current state is the culmination of all the events up to the given point in time.
- 14. streamnative.io Event Based Data Model
- 15. Complete Rebuild We can discard the application state completely and rebuild it by re-running the events from the event log on an empty application. Temporal Queries We can determine the application state at any point in time. Notionally we do this by starting with a blank state and rerunning the events up to a particular time or event. Event Replay If we find a past event was incorrect, we can compute the consequences by reversing it and later events by replaying the new event and later events. Correcting Bad Data/Code If a code update introduces an error into the data, e.g., bad calculation. We can correct the data by replaying the data through a “good” version of the code. Event Reversal This helps account for the lack of distributed transactions across microservices. The reversal of an event to "add $10 to account 123", would be "subtract 10$ from account 123" Event Sourcing Benefits
- 17. Apache Pulsar is a Cloud-Native Messaging and Event-Streaming Platform.
- 18. Pulsar Pub/Sub Model Producer Consumer Publisher sends data and doesn't know about the subscribers or their status. All interactions go through Pulsar, and it handles all communication. Subscriber receives data from publisher and never directly interacts with it. Topic Topic
- 19. Unified Messaging Model Simplify your data infrastructure and enable new use cases with queuing and streaming capabilities in one platform. Multi-tenancy Enable multiple user groups to share the same cluster, either via access control, or in entirely different namespaces. Scalability Decoupled data computing and storage enable horizontal scaling to handle data scale and management complexity. Geo-replication Support for multi-datacenter replication with both asynchronous and synchronous replication for built-in disaster recovery. Tiered storage Enable historical data to be offloaded to cloud- native storage and store event streams for indefinite periods of time. Pulsar Benefits
- 20. Why ApachePulsarfor Event Sourcing? Event Streaming Platform Guaranteed Message Delivery Resiliency Infinite Scalability
- 21. Apache Quarkus is is a full-stack, Kubernetes- native Java framework.
- 22. Container First Amazingly fast boot time, incredibly low memory offering near instant scale up and high-density memory utilization in container orchestration platforms like Kubernetes. Unifies imperative and reactive Combines both the familiar imperative code and the reactive programming styles when developing applications. Community & Standards Provides a cohesive, full-stack framework by leveraging a growing list of best-of-breed libraries that you love and use. All wired on a standard backbone. Cloud Native The combination of Quarkus and Kubernetes provides an ideal environment for creating scalable, fast, and lightweight applications. Developer Joy A cohesive platform for optimized developer joy with unified configuration and no hassle native executable generation. Zero config, live reload in the blink of an eye.. Quarkus Benefits
- 23. Why Quarkus for Event Sourcing? Reactive Programmingin Java Containerfirst Philosophy Kubernetes Native Optimizedmemory usage
- 24. Implementation
- 25. streamnative.io Event Sourcing Application Architecture
- 26. streamnative.io Update Driver Position ● Mobile app automatically sends position update every 30 seconds with this command. ● Command handler generates a new “Driver location” event in response to this message. ● Command handler performs additional data enrichment.
- 27. streamnative.io View #1: Assigned Driver Location ● Once a driver has been assigned to a customer, we want to track the driver in route. ● View comprised of the driver’s most recent positions, which can be displayed visually as a map. ● Solution: Use an exclusive subscription and filter by driver id
- 28. streamnative.io View #2: Analytics ● To help optimize driver placement, a heat map of driver density per “grid” is calculated. ● View is a map of grid_id’s and number of drivers located within that grid. ● Solution: Use the Reader interface, go back N minutes and recalculate.
- 29. Demo Commands & Views Code: git@github.com:david-streamlio/GottaEat-UI.git
- 30. streamnative.io Key Takeaways ➔ Event Sourcing is great for storing data that needs to be consumed in different ways. ➔ Apache Pulsar is a great option for event-based storage. ➔ Pulsar’s API can be used to create materialized views of event data.
- 31. streamnative.io [Webinar] Building Microservices Watch Now Learn More [Blog post] Event-Driven Microservices
- 33. Hosted by Save Your Spot Now Use code SUMMIT20 to get 20% off. Pulsar Summit San Francisco Hotel Nikko August 18 2022 Summit Highlights: ● 5 Keynotes ● 12 Breakout Sessions ● 1 Amazing Happy Hour Speakers from:
- 34. Let’s Keep in Touch! David Kjerrumgaard Developer Advocate @davidkjerrumga1 https://www.linkedin.com/in/davidkj/ https://github.com/davidkj
- 35. Q&A