3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)
0 likes132 views
The document outlines strategies for implementing a flexible and cost-effective cloud architecture using streaming data with Apache Kafka. It emphasizes starting with low-risk, high-value use cases, understanding data flows, and gradually scaling as needed, while noting the limitations of partitions in Kafka. Additionally, it discusses the importance of end-to-end visibility and monitoring for successful data pipeline management.
![Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
Do Nothing
Elastic Scaling in SaaS: Confluent Cloud
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
1 Click—Select CKU from drop
down in cluster management UI and
click Apply Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica](https://image.slidesharecdn.com/cloudconfabeventsmay2020-future-proofeventstreaminganz1-200528212721/85/3-Ways-to-Deliver-an-Elastic-Cost-Effective-Cloud-Architecture-ANZ-17-320.jpg)
3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture (ANZ)
- 1. 3 Ways to Deliver an Elastic, Cost-Effective Cloud Architecture Guru Sattanathan Solution Engineer Confluent (Melbourne)
- 2. Mark Teehan Sr. Solutions Engineer Confluent John Kriter Senior Technical Architect within Emerging & Growth Accenture Guru Sattanathan Solutions Engineer Confluent A little bit about us 2
- 3. Main themes for today 3 Scale Quickly Deploy Anywhere Run efficiently Faster Better Cheaper
- 4. Overview of the streaming pipeline 4 Fast Data Slow Data Agents StreamsSources Processing / Transform Storage & Presentation Use Cases: Streaming ETL Aggregation CDC Data Sync Logs
- 5. Start at the end and work backwards 5 Start at the end and work backwards
- 6. Fantasy Zone Finding the value sweet spot Find an initial use case that is valuable, but not absolutely critical Avoid low value high risk use cases at first - you want to understand the tech and the way you’ll use it first Nothing exists in the high value low criticality zone, so don’t bother looking too hard there Be aware that the opportunity zone borders both the fantasy and danger zones! 6 Value Criticality Danger Zone Opportunity Zone
- 7. Follow a single thread all the way through Information flows are rarely as simple as you think they are Take the time to trace one flow end to end You can replace just part of a flow at first You can also branch a flow 7
- 8. 8 A chain is only as strong as its weakest link Your data pipeline is a chain
- 9. Kafka can be your elastic buffer Plan for resiliency or elasticity in your pipeline But don’t over optimize for it Everyone thinks they’re gonna get millions of messages per second 9
- 10. Understanding what is happening is critical 10
- 11. A very brief guide to Kafka A cluster is… well, a cluster (N brokers) Brokers are servers that make up a cluster (they do the actual work) Topics are logical constructs that consist of 1+ partitions Partitions live on brokers As data is written to a topic it is appended to one partition Normally partitions have a leader and several followers who keep copies of data for resiliency 11
- 12. 12 Our cloud provides aggregated metrics in the UI and via a REST API so you can see what is happening live
- 13. 13 Consumer Lag is also a powerful monitoring tool
- 14. Avoiding architectural bottlenecks Understanding the nature of Apache Kafka Partitions • The unit of work in Kafka • The unit of scale and organization for your data Ordering • Order only exists within partitions • Providing global order is generally not worth it Data placement (proximity) • Making your stream (Kafka) fast is only part of the story • You need to ensure that your processing can / will be fast 14
- 15. Dataflow A graphical tool to help you quickly locate issues across your streaming applications End to end visibility helps you find where and when things break. Visibility into: ● Producers ● Topics ● Partitions ● Consumers & Consumer Groups 15
- 16. 16 Understanding when to scale is as important as being able to scale Plan for both
- 17. Confluent Cloud Milliseconds Minutes Basic, Standard [0-100Mbps] Do Nothing Elastic Scaling in SaaS: Confluent Cloud *Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already. Dedicated [Mbps - Gbps] 1 Click—Select CKU from drop down in cluster management UI and click Apply Changes Other Kafka Services Days - Weeks Determine how much capacity is needed Procure capacity* Configure new brokers a. Disks b. OS c. Network d. Kafka (application) Identify partitions on specific brokers to rebalance & topics they are part of For each Topic: migrate partitions a. Increase ISR +1 b. Wait for new replica to sync c. Failover master d. Reduce ISR -1 e. Delete old replica
- 18. Start small and grow Incremental investment provides lower risk • You will learn as you go • It’s easier to adjust when the scope is smaller Iterate quickly • Try many lightweight approaches • Pick the one that works best Do not be distracted by the long term • You may not be right the first time • Don’t risk it all on one big bet 18
- 19. Stay within the lines… the lines are our friends
- 20. Recommended limits for Partitions A single partition has a finite limit We recommend • No more than 5 MBps per partition Ingress (write) • No more than 15MBps per partition Egress (read) We list or limits and recommendations publicly: https://docs.confluent.io/current/cloud/features/cluster-types.html#dedicated-cl uster-ckus-and-limits Please read them! 20
- 21. Deploy anywhere
- 22. Confluent Cloud: consistent experience on any provider Same Experience • UX / CLI • Tiers / cluster types Same services • Kafka • ksqlDB • Connect • Schema Registry Fully managed • Pure SaaS, no infra to manage 22
- 23. You can have clusters from many providers in the same account 23
- 24. Show me! Confluent Cloud UI
- 25. Confluent Cloud and Confluent Platform Everything we learn in Confluent Cloud goes into Confluent Platform so you can have the same type of experience you get from our cloud anywhere you need. We use this tech ourselves: Confluent Operator for K8s is how we multi-cloud 25 Confluent Platform Confluent Cloud
- 26. Key takeaways Start small Keep the entire flow in mind Understand the end state: start at the end and work backwards Know when and how to scale, but don’t over provision early / unnecessarily Run where you want to and where it makes the most sense 26
- 28. Hybrid streaming Bridging OT and IT • Safely bridge with one way flows Linking remote or satellite locations intelligently • Upload aggregate data in real-time, but full data in anomaly conditions • Reducing connectivity costs Streaming data in real-time instead of batching at night • Reducing time to value 28
- 29. Augmenting critical legacy systems Enterprises have decades of software and processes in existing platforms Frontending legacy systems is common already Before we used queues Ever try to replay a queue? Kafka & Confluent won’t replace your mainframe or SAP, but they can help you get more from them 29 Connect Upstream Data
- 30. Data distribution: information services Lower cost data distribution Replayable syndication streams Reversing Push and Pull models Deferring costs from service provider to service consumer Ex. Pricing information is an information service 30 Customer A Customer B Customer C Organizational Boundary
- 31. Thank you! Mark Teehan teehan@confluent.io Rahul Natarajan Rahul.Natarajan@accenture.com cnfl.io/meetups cnfl.io/slackcnfl.io/blog