Microservices in a Streaming World
Download as PPTX, PDF1 like490 views
The document discusses the evolution of microservices in modern distributed applications, tracing their roots from monolithic architectures and service-oriented architecture (SOA) to their contemporary form, emphasizing the advantages of decoupling services. It highlights the role of Apache Kafka as a backbone for data-centric architectures, facilitating service communication, persistence, and scalability. The document also notes the complexities and challenges inherent in microservices, particularly regarding dependency management and state maintenance, which Kafka addresses through its unique messaging and stream processing capabilities.
![What’s different about Kafka? Topic Partitions and Keyed Messages
- Topics/Queues are not the smallest unit of
scalability
- Topics partitions are distributed across
brokers for parallel in-order consumption
- This is very different from a cluster of
traditional message brokers
- [graphic of topic partitions with parallel
Producers, Brokers, and Consumers]
- Sometime you can just use more keys
instead of more topics
- Eg. don’t create a new topic for every user,
or IoT device, create unique keys
- This is proven to scale to many millions of
connected users, cars and IoT devices
- [graphic to show Keyed messages get
distributed across topic partitions]](https://image.slidesharecdn.com/microservices-deck-hans-australia-190219072320/85/Microservices-in-a-Streaming-World-80-320.jpg)
Microservices in a Streaming World
- 1. Microservices in a Streaming World Modern distributed applications and infrastructure
- 2. About me Hans Jespersen hans@confluent.io https://www.linkedin.com/in/hansjespersen https://github.com/hjespers U of Waterloo – Punch Cards, COBOL, Assembler, RJE, Screen Scraping AT&T – Unix, Client/Server, Tuxedo & Transactions, Pre-Web Internet Sun – Solaris, ONC RPC, CORBA TIBCO – Yahoo!Quotes, Pub/Sub, Multicast, JMS, ESB, SOA, WS-* Solace – Messaging HW, MQTT ,AMQP, REST Confluent – Kafka, Event Stream Processing Personally - active in IoT, open source, MQTT
- 3. Not going to talk about... Meaningless, vendor marketing, definition of Micro Services
- 4. Let’s parse this out together…
- 5. Love Martin Fowler’s Work
- 7. https://youtu.be/_RgUxUTuxH4 – or Google “Kafka Summit Martin Fowler”
- 8. I am going to talk about... Patterns in Modern Distributed Architecture and Application Infrastructure
- 9. You’ve probably heard of Microservices.
- 10. But what are they really?
- 11. Are they SOA all over again? SOA Microservices
- 12. Or are they an evolution? SOA
- 13. We used to build monolithic applications.
- 14. These applications came in a variety of different forms.
- 15. Companies built lots of them.
- 16. But the teams in these companies were grouped into business areas. Silos.
- 17. So the applications would also form into silos along their business lines.
- 18. This led to a lot of duplication across different applications, in different silos.
- 19. The duplication made the IT function expensive to run.
- 20. Then someone had an idea.
- 21. What if they pulled out the common pieces into reusable “services”? Service
- 22. Less code in each silo seemed like a good idea. Service
- 23. Then, they realized they could build whole companies based on these reusable services.
- 24. This seemed like a very good idea!
- 25. The trick was to rearrange the people to match the services.
- 26. So instead of sitting in silos…
- 27. The teams would match the architecture that they wanted to end up with.
- 28. SoA This pattern was termed a ‘Service Oriented Architecture.’
- 29. But - there was a problem.
- 30. Refactoring companies turned out to be slow and difficult. Services
- 31. So people took the same pattern designed for big companies…
- 32. …and applied it to break up monolithic applications...
- 33. …knowing that with this architecture, the companies would be able to grow…
- 34. …and grow…
- 35. …and grow.
- 36. This was termed Microservices: Change through small, well-defined services that are easy to reuse.
- 37. So silos were broken down…
- 38. …and monoliths were broken up.
- 39. Evolving incrementally towards contemporary service estates.
- 40. But things were not entirely rosy yet.
- 41. While silos seemed expensive and wasteful, they had one big advantage...
- 42. Each application was free to handle change independently of the applications around it.
- 43. Every application was, in some sense, an island.
- 44. Microservices are built with HTTP, REST, or some other protocol made from requests and replies. Request Reply
- 45. This works well when ecosystems are small… Buying Widgets
- 46. But gets harder as they grow more complex and more interconnected.
- 47. Services are tightly coupled. No islands here!
- 48. So if one service fails…
- 49. …or even just runs slowly… Buy Widgets
- 50. The fall-out could be much larger.
- 51. Others end up feeling that pain. Buy Widgets
- 52. Yet, at company scale, the majority of processes run in the background anyway. They are asynchronous to one another. Online Billing Inventory Fulfillment Fraud Offline
- 53. So it makes sense to DECOUPLE services from one another. Billing Inventory Fulfillment Fraud Offline Online Decouple
- 54. Apache Kafka™ helps with this as it provides a data backbone for your services. Billing Inventory Fulfillment Finance Fraud HTTP etc Offline Online
- 55. This connects services together.
- 56. It also connects their DATA together. All Your Data
- 57. The three tenets of messaging… embedded into a layer of permanence. Decoupling Notification Data Transfer Permanence
- 59. So every service gets the data it needs. All Your Data
- 60. Unlike typical service frameworks, it also DECOUPLES services from one another, so they can evolve independently.
- 61. This makes it easier to move away from legacy architectures, to evolve away from the past… All Your Data LEGACY
- 62. …towards a better-factored future, whatever that may look like. All Your Data LEGACYNew Services Analytics
- 63. So wherever your business ends up… Cloud Another Device Another Geography
- 64. Apache Kafka provides the Service Backbone built to handle today’s data-centric world... Big Data Ready
- 65. In a way that can adapt to your company's future, wherever you might take it.
- 66. Services built on the POWER and IMMEDIACY of an Event Streaming Platform. Event Streaming Platform
- 68. So what is this Kafka thing you speak of so highly?
- 69. How do you see Kafka?
- 70. Traditional Messaging Functionality Decoupling of Producers and Consumers
- 71. Message Exchange Patterns (MEP) Topic = Publish/Subscribe - N of N delivery Queues = Point-to-Point - 1 of N delivery
- 72. Message Exchange Patterns (MEP) Request/Reply Content-Based Router
- 73. Message Delivery Semantics At-Most-Once “Best effort” “Reliable” QoS 0 At-Least-Once “Guaranteed” “Certified” QoS 1 Exactly Once “Once-and-Only-Once” “Transactional” QoS 2
- 74. Wide Spectrum of Messaging Offerings Ultra- low Latency (often no broker in the middle) High Volume (Persistent or Non-Persistent) Highly Available (Clustered and Fault Tolerant) Embedded Messaging (inside apps) Cross Datacenter / Organizational / B2B Enterprise Message Bus Messaging-as-a-Service Web / IoT Messaging Instant Messaging
- 75. “publish-subscribe messaging rethought as a distributed commit log”
- 76. Kafka is a Mashup Mashup of some well proven concepts into something even greater and easier to use: EAI + ETL Messaging Middleware + Big Data Batch + Real-time Data Movement + Data Processing Log Data Streams + Structured Database Tables
- 77. + Distributed clustered storage Kafka is a blend of messaging, stream processing, ETL and modern database designs built around a distributed log + Streaming platform Pub/Sub Messaging ETL Connectors Spark Flink Beam IBM MQ TIBCO RabbitMQ Mulesoft Talend Informatica Kafka is much more than messaging + Exactly Once + Designed for the Cloud + Inter DC replication + Schema evolution Stream Processing Confluent Confidential
- 78. What’s different about Kafka? Topics are also Queues Consumers can share one copy of the data • Independent consumers share the same log • Inter-dependent consumers share the same log • No need for Topic/Queue bridging or multiple copies of the data Message processing is greatly simplified - There is no “head’ of the queue - Writes are sequential, distributed, and parallel
- 79. What’s different about Kafka? Messages are not deleted when consumed Messages in the commit logs are persistent and immutable Slow Consumers are (very) decoupled from Fast Producers Batch and real-time are unified Message Replay, Replication, and Auditing are built-in (for free) All production messaging deployment need some form of these Message Retention is not a waste of disk space You need to size for offline/disconnected consumers anyway Distributed State can always be recreated from a common commit log Makes distributed HA apps much easier to build
- 80. What’s different about Kafka? Topic Partitions and Keyed Messages - Topics/Queues are not the smallest unit of scalability - Topics partitions are distributed across brokers for parallel in-order consumption - This is very different from a cluster of traditional message brokers - [graphic of topic partitions with parallel Producers, Brokers, and Consumers] - Sometime you can just use more keys instead of more topics - Eg. don’t create a new topic for every user, or IoT device, create unique keys - This is proven to scale to many millions of connected users, cars and IoT devices - [graphic to show Keyed messages get distributed across topic partitions]
- 81. From an event stream / transaction log we can derive all of the following database centric features: - Replication - Secondary Indexing - Caching - Materialized Views What’s different about Kafka? Duality of streams and databases Duality of a message streams and database tables is a key design point =
- 82. (Good) Microservices avoid shared mutable state Shared, mutable state
- 83. Old World: REST Based Microservices Interconnect GUI UI Service Order s Returns Pay Fulfilment Stock Confluent Confidential Each Microservice has to maintain their own stateful nature by using their own databases 1. Difficult to Enforce Same REST API standards across many languages and micro-services. 2. Rest APIs Inherently Slow: Limited to Thousands calls/sec. 3. Inter Service Dependencies are Messy. 4. Each Service Needs to Maintain State. 5. Difficult to enforce consistent security standards. 6. Logging is distributed between services. 7. Version compatibility between services is difficult.
- 84. Streaming Microservices with Kafka GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Confluent Confidential Database Sources Now Centralized on the Kafka Bus for all microservices 1. Service inter-communication standard enforce by Kafka Schema Registry. 2. Millions of messages per second on cheap hardware. 3. No Inter-Service Dependency: just depend on Kafka. 4. Each service can be stateless: Kafka maintains state. 5. Security can be enforced by ACLs from Kafka. 6. Logs can be aggregated into Kafka. 7. Version compatibility can be enforced by Scheme Registry. 8. Kafka is inherently HA, horizontal scalable: still no central point of failure.
- 85. What’s different about Kafka? Ecosystem and Adoption The Kafka ecosystem is flourishing and developer adoption continues to grow • Confluent Platform additions (REST Proxy, Schema Registry, KSQL etc.) • Third Party Connectors ( Confluent Hub) • Open Source contributions from individuals, corporations, vendors, consulting organizations • Inside and outside of Big Data/Stream Processing
- 86. Adoption of Event Streaming 60%Fortune 100 Companies Using Apache Kafka
- 87. Event Streaming at the Heart of the Enterprise
- 90. Embrace a Distributed World