Voxxed Athens 2018 - Eventing, Serverless, and the Extensible Enterprise
Download as PPTX, PDF0 likes467 views
The document discusses event-driven application architectures on Microsoft Azure. It provides an overview of Azure messaging and eventing services like Event Hubs, Service Bus, Event Grid, and IoT Hub. It also discusses how serverless architectures on Azure using functions can enable reactive and event-driven applications. Building and device management scenarios are provided as examples of how sensor data from IoT devices could be processed and reacted to based on events.
Voxxed Athens 2018 - Eventing, Serverless, and the Extensible Enterprise
- 1. Clemens Vasters Product Architect – Azure Messaging @clemensv OASIS AMQP Technical Committee Chair CNCF CloudEvents Spec Owner OPC UA PubSub Co-Architect Eventing, Serverless, and the Extensible Enterprise
- 2. Event Hubs Telemetry stream ingestion Kafka/AMQP Service Bus & Azure Queues Cloud messaging Event Grid Event distribution Logic Apps Workflow and LOB Integration IoT Hub IoT messaging and manage- ment Notification Hubs Mobile push notifications Relay Discovery, Firewall/NAT Traversal Stream Analytics Data Lake Storm Spark Azure Functions Storage Blobs Microsoft Azure
- 4. Event-Driven Photos Blob Container “uploads” blobCreated Cloud News Website Editorial Office (edit/publish) Mobile (out shooting) Uploads RAW and/or JPG Blob Container “sizedImages” Normalize format, auto-classify, tag, move Blob Container “inventory” read/delete aircraft, indoors, blue, white, UK, Cosford, Canon, 6D blobCreated Create sized images photoIngested read Index Database aircraft, indoors, blue, white, UK, Cosford, Canon, 6D sizedImageCreated write Lightroom Photoshop Newsroom App Check & add tags New! Import? import edit photoReindexed update update republish blobCreated blobDeleted Aircraft in Trouble!!! Update published assets publishDone
- 5. Building Management Device Management • Climate • Energy • Lighting • Water • Gas • Space Utilization • Access Control • Movement Tracking • Elevator/Escalator • Refrigeration • Networking • Fire/Smoke/Gas • Biohazards • Flooding/Wind • Earthquake • Power • Battery • Temperature • Vibration • Component Health … • Firmware • TPM • Hypervisor • OS • Container • App • Config • Credentials • Access Control • Traces • Logs • Data … Sensor-Event-Driven Building Management: Two Perspectives
- 7. Room Unit Floor Building Site Gas/Biohaz Climate Fire/Smoke Occupancy Org /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/occupancy /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/fire /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/gasbio /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/climate /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4 /org/contoso/site/berlin/building/41/floor/3/unit/j /org/contoso/site/berlin/building/41/floor/3 /org/contoso/site/berlin/building/41 /org/contoso/site/berlin/ /org/contoso/ Building Management Some Analytics Questions: • Are there people in room 41 3J/4? • What room is unoccupied in building 41? • Is there a fire alarm at the site? • How is the air quality on the lab floor? • What’s the temp in tenant unit 41 3J? Some Reactive Actions: • Signal evacuation and alert the Fire Department if any fire or gas/biohaz sensor on site goes into an alert state. • Adjust floor HVAC when average temp on any building floor deviates by +/- 2C from 20C. • Alert Security when unexpected occupancy is detected in Unit 41 3J.
- 8. Room Unit Floor Building Site Gas/Biohaz Climate Fire/Smoke Occupancy Org /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/occupancy /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/fire /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/gasbio /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/climate /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4 /org/contoso/site/berlin/building/41/floor/3/unit/j /org/contoso/site/berlin/building/41/floor/3 /org/contoso/site/berlin/building/41 /org/contoso/site/berlin/ /org/contoso/ Building Management“Are there people in room 41 3J/4?” • SUBSCRIBE /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/occupancy • EVENT • Topic: /org/contoso/site/berlin/building/41/floor/3/unit/j/room/4/sensors/occupancy • Data { sensor: xyz, bldg: 41, room: “3J/4“, persons : 0 } What room is unoccupied in building 41? • SUBSCRIBE /org/contoso/site/berlin/building/41 • EVENT • Topic: /org/contoso/site/berlin/building/41 • Filter on subject: /floor/*/unit/*/room/*/sensors/occupancy == 0 • Data { sensor: xyz, bldg: 41, room: “3J/4“, persons : 0 } Is there a fire alarm at the site? • SUBSCRIBE /org/contoso/site/berlin • EVENT • Topic: /org/contoso/site/berlin • Filter on subject: /building/*/floor/*/unit/*/room/*/sensors/fire (type== ‘alarm’) How is the air quality on the lab floor? • SUBSCRIBE /org/contoso/site/berlin/building/41/floor/1 • EVENT • Topic: /org/contoso/site/berlin/building/41/floor/1 • Filter on subject: /unit/*/room/*/sensors/climate What’s the temp in tenant unit 41 3J? • SUBSCRIBE /org/contoso/site/berlin/building/41/floor/3/unit/j • EVENT • Topic: /org/contoso/site/berlin/building/41/floor/3/unit/j • Filter on: /room/*/sensors/climate
- 9. Serverless and Microservices (Spoiler: Of course there are servers) (Spoiler: This isn’t new)
- 10. • “Serverless” is PaaS • The promise of “Platform as a Service” is to liberate you from managing low- level infrastructure and focus on apps and data • Virtualization is IaaS • Containers and container orchestrators are a very flexible way to share, configure, and reuse portions of virtualized infrastructure. • Container platforms may form a complete virtualization layer, including software-defined networking. “Serverless” = Platform as a Service (PaaS) “Serverless” means you supply the app Containers may be how “you manage”
- 11. • Managed Cluster • You supply applications that are deployed on a cluster that allows for placement, replication, ownership consensus and management of stateful resources • Managed Middleware • You supply applications that are deployed on sets of independent, “stateless” middleware servers, like web servers or pure compute hosts • Applications may be always-on or start on demand; typically maintain shared cached state and resources • Managed Functions • You supply (small groups of) function implementations that are run for you when triggered by a configured condition. Degrees of Serverlessness
- 12. • … is responsible for holding, processing, and/or distributing particular kinds of information within the scope of a system • … can be built, deployed, and run independently, meeting defined operational objectives • … communicates with consumers and other services, presenting information using conventions and/or contract assurances • … protects itself against unwanted access, and its information against loss • … handles failure conditions such that failures cannot lead to information corruption A “Service” is software that …
- 13. • Defining property of services is that they’re Autonomous • A service owns all of the state it immediately depends on and manages • A service owns its communication contract • A service can be changed, redeployed, and/or completely replaced • A service has a well-known set of communication paths • Services shall have no shared state with others • Don't depend on or assume any common data store • Don't depend on any shared in-memory state • No sideline communications between services • No opaque side-effects • All communication is explicit • Autonomy is about agility and cross-org collaboration Services: Autonomous Entities
- 14. The modern notion of “Service” is not about code artifact counts or sizes or technology choices. It’s about ownership.
- 15. • A service can be made up of a fleet of independently deployed functions that jointly operate on a shared set of resources • The service interface is made up from the union of the function interfaces • The function interfaces may be a mix of RPC-style call interfaces and event driven ones Microservices and Functions-as-a-Service? Function API and Implementation
- 16. • Independently deployable and versionable • Independently scalable • Can have independent communication paths and invocation triggers • Webhook • Queue • DB Transaction • Etc. Upsides of Functions? Function API and Implementation
- 17. • A system is a federation of services and systems, aiming to provide a composite solution for a well-defined scope. • The solution scope may be motivated by business, technology, policy, law, culture, or other criteria • A system may appear and act as a service towards other parties. • Systems may share services • Consumers may interact with multiple systems System
- 19. • “REST“ is great for interactively accumulating and acting on state from multiple sources. • Let‘s not pretend all clients are like that – there‘s a lot more • HTTP and RPC are great to obtain immediate answers. • The longer it takes to generate the answer, the more brittle the model becomes Service Communication
- 20. Command Query Job Handover Report Measurement Notification Request Assignment Update Transfer Trace
- 21. Command Query Job Handover Report Measurement Notification Request Assignment Update Transfer Trace
- 22. Intents Facts
- 23. Intents Facts Messaging Expectations Conversations Contracts Control Transfer Value Transfer History Context Order Schema Eventing
- 24. Messaging Eventing A B C A ? ?
- 25. Device Cloud Gateway Device Queues ! ! ! ! Backend Process Azure IoT Hub Azure Service Bus < > < > Messaging – Example: Device Command
- 26. Backend Process Azure Logic Apps Azure Service Bus Workflow Manager Step Handler Step Handler Step Handler Step Handler Accounting and Stuff Messaging – Example: Workflow Execution
- 27. Discrete Series Events Independent Report State Change Actionable Time Ordered Context Partitioned Analyzable Report Condition
- 28. • Report independent, actionable state changes to authorized subscribers • “Blob created” • “Sales lead created” • “Order confirmed” • Allows simple, noninvasive, reactive extension of the functionality of a service Discrete Events are an Extensibility Enabler
- 29. Device Collect Readings Ingestor Read Partition Example: Data Series Processing 🕓 Analyzer “Pull” flow towards stateful instance Stateful instances and Actors Azure Event Hubs
- 30. Device Route Alarm ! Distributor Subscrip- tions ! ! ! ! ! L B Example: Discrete Event Handling –Alarms Handlers “Push” flow towards stateless handlers “Serverless” Functions Azure Event Grid
- 31. • Event Protocol Suite developed in CNCF Serverless WG • Common metadata attributes for events • Flexibility to innovate on event semantics • Simple abstract type system mappable to different encodings • Transport options • HTTP(S) 1.1 Webhooks, also HTTP/2 (v0.1) • MQTT 3.1.1 and 5.0 (draft) • AMQP 1.0 (draft) • Encoding options • JSON (v0.1, required for all implementations) • Extensible for binary encodings: Avro, MessagePack, AMQP, etc. CNCF CloudEvents?
- 32. First-class support for CloudEvents on Azure Event Grid • CloudEvents v0.1 support for all Azure platform events, available in production • CloudEvents v0.1 support for Event Grid custom topics (structured JSON encoding) • Docs: https://aka.ms/egcncfdocs Event Hubs Resource Groups Azure Subscriptions Custom Topics Blob Storage Azure Functions Azure Automation WebHooks Logic Apps Event HubsService Bus IoT Hub Storage (GPv2) CloudEvents Sources Custom Topics support CloudEvents input Natively publish any event in CloudEvents format
- 33. • “Core” functions of services allow for direct interaction: • Commands, Requests, etc. • Extensions react to events emitted by services based on core activities. • Might turn to the emitting service to ask for details or perform actions. Event-Driven: Cores + Extensions Direct Interaction Extension by subscription Messaging Eventing
- 34. • Event handlers may need to be implemented by apps that reside out of easy network reach • Behind NATs • Inside of Containers w/o public endpoints • A: Route via queues • B: Use a relay Extend to the Edge? Extend into containers? NAT? Container?
- 35. Azure Relay HTTP • Regular Node.js Listener var http = require('http‘); var port = process.env.PORT || 1337; http.createServer(function (req, res) { res.writeHead(200, { 'Content-Type': 'text/plain' }); res.end('Hello Worldn'); }).listen(port); • Relayed Node.js Listener var http = require('hyco-https‘); http.createRelayedServer( cxnString, function (req, res){ res.writeHead(200, { 'Content-Type': 'text/plain‘ }); res.end('Hello Worldn‘); }).listen(); http://aka.ms/vxa
- 36. • Services: Autonomous software entities grouped around resource ownership and team ownership scopes • Platform-as-a-Service: Hosting options for services with tailored degrees of control for certain scenarios. • Managed clusters: Complex, stateful, high reliability, always-on. • Managed servers: Stateless/shared-nothing, on-demand start. • Managed functions: Event-driven, short-lived • Messaging and Eventing: Communication backplane for services, eventing as extensibility enabler. Summary
- 37. Event Hubs Telemetry stream ingestion Service Bus & Azure Queues Cloud messaging Event Grid Event distribution Logic Apps Workflow and LOB Integration IoT Hub IoT messaging and manage- ment Notification Hubs Mobile push notifications Relay Discovery, Firewall/NAT Traversal Stream Analytics Data Lake Storm Spark Azure Functions Storage Blobs Microsoft Azure
- 38. © 2017 Microsoft Corporation. All rights reserved. The text in this document is available under the Creative Commons Attribution 3.0 License, additional terms may apply. All other content contained in this document (including, without limitation, trademarks, logos, images, etc.) are not included within the Creative Commons license grant. This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes. This document is provided "as-is." Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it. Some examples are for illustration only and are fictitious. No real association is intended or inferred. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Editor's Notes
- #11: How to decide Does app require third party components? Do security requirements mandate the use of third party monitoring software? Does app require specific framework configurations? The more customization that is required, the more one moves towards IaaS and away from PaaS.