Bridge Your Kafka Streams to Azure Webinar
Download as PPTX, PDF1 like429 views
The document presents an overview of bridging Kafka Streams to Azure Managed Apache Kafka service, highlighting its benefits for businesses in adapting to digital commerce trends. It discusses Confluent's fully-managed Kafka solutions, features, and the advantages of using Azure Marketplace for seamless integration and billing. The text emphasizes the significance of real-time data processing in driving modern applications and provides insights into various use cases and flexible cloud offerings.
Bridge Your Kafka Streams to Azure Webinar
- 1. Bridge your Kafka Streams to Azure Managed Apache Kafka® Service on Azure
- 2. Introductions Joanna De Castro- Dee APAC ISV Partner Development Manager Microsoft Ben Echols Senior Product Manager, Confluent Derk Van Ogtrop BD Alliances Director APAC, Confluent
- 3. The Microsoft Marketplace Confluent introduction Fully managed Kafka on Azure Demo Kafka clusters for every need Next Steps Agenda
- 4. The Microsoft Marketplace Joanna Dee APAC Partner Development Manager, Global ISVs September 23, 2020
- 5. ©Microsoft Corporation Test drive innovation on Microsoft AppSource and Azure Marketplace https://player.vimeo.com/video/437212098
- 6. ©Microsoft Corporation The way we all do business is changing… 67% of business buyers would prefer to gather their own information online and not interact with a sales rep Business-to-business e-commerce is forecasted to compound by 10% annually through 2023 Sources: Forrester, May 2019 Quantum Metric, March 2020 e-commerce has experienced an average revenue weekly growth rate increase of In the last few weeks, 52% and an 8.8% increase in conversion rates
- 7. ©Microsoft Corporation Microsoft AppSource Azure Marketplace The commercial marketplace delivers the best apps 24/7 Category and workload building blocks for Azure Industry-focused line of business Productivity applications & Publish your app to surface in…
- 8. ©Microsoft Corporation Apps in key categories and industries MigrationAI/Analytics Data SecurityIoT ManufacturingFinancial Services Healthcare RetailInsurance
- 9. © Copyright Microsoft Corporation. All rights reserved. Thank you.
- 10. Confluent Snapshot | Founded September 2014 10 founded by the Original Creators of Apache Kafka® 67% of the Fortune 500 uses Apache Kafka Kafka is in Production at 200,000+ Companies Kafka was invented at LinkedIn & Donated to the Apache Foundation Confluent was LinkedIn’s 1st Corporate Investment
- 11. JP Morgan Chase’s 2019 “Hall of Innovation” Award Google’s 2019 &’20 “Technology Partner of the Year” Data & Analytics # 10 Forbes Cloud-100 https://www.credit-suisse.com/about-us-news/en/articles/media-releases/credit-suisse-ag-announces-disruptive-technology-recognition-award-winners-201912.html Confluent is The Leading Event Stream Processing Technology
- 12. Open Source Kafka Adoption is Accelerating 12
- 13. Every Organization Uses a Variety of Software Systems Data Diode
- 14. Kafka Has Become a Common Way Organizations Connect those Systems
- 15. Central-Nervous System for Data – Driving Cloud App/Dev Hadoop ... Device Logs ... App ...MicroserviceMainframes Azure Data Services Splunk ... Data Stores Logs 3rd Party Apps Custom Apps / Microservices Same Day Transactioning (Account Open) Fees Charges & Billing Real-time Customer 360 Machine Learning Models Real-time Data Transformation Real-time Fraud Detection
- 16. Build an event streaming platform, and put it at the heart of every company.
- 17. Confluent Cloud Fully-managed Born in the cloud: no infrastructure Guaranteed 99.95% uptime SLA Self-serve for GBps Complete platform Connect, ksqlDB, Schema Registry Capable of end-to-end applications Kafka from the people who made it Run anywhere Global availability on AWS, Azure, and GCP Bridge on-prem to cloud with hybrid Kafka Extend streaming apps across clouds Security & compliance Control access to clusters and resources Authenticate users and applications Protect data with end-to-end encryption Flexible Cluster types for all scenarios & sizes Many networking options Customer choice of features Elastic Scale in-place w/o downtime No storage limits
- 18. What does Fully Managed really mean?
- 19. Confluent Cloud is truly SaaS On-Premises IaaS (Infrastructure as a Service) PaaS (Platform as a Service) SaaS (Software as a Service) Applications Applications Applications Applications Data Data Data Data Runtime Runtime Runtime Runtime Middleware Middleware Middleware Middleware O/S O/S O/S O/S Virtualization Virtualization Virtualization Virtualization Servers Servers Servers Servers Storage Storage Storage Storage Networking Networking Networking Networking You Manage Fully Managed Open Source Kafka Confluent Cloud Hosted Kafka Proprietary Services (Kinesis)
- 20. What does a complete platform mean?
- 21. Kafka | Schema Registry | KSQL | Connectors | Auto data balancer | | Replicator | MQTT Proxy | REST Proxy | Connectors | Database changes Log events IoT events Web events Transformations Custom apps Analytics Monitoring Hadoop Database Data warehouse CRM Fully managed service Available self managed components DATA INTEGRATION REAL-TIME APPS Complete Platform | Apache Kafka re-engineered for cloud
- 22. Kafka | Schema Registry | KSQL | Connectors | Auto data balancer | | Replicator | MQTT Proxy | REST Proxy | Connectors | Database changes Log events IoT events Web events Fully managed service Available self managed components DATA INTEGRATION REAL-TIME APPS Complete Platform | Apache Kafka re-engineered for cloud
- 23. Azure Marketplace Sign-up directly through the Azure portal Use Azure as your payment method Get full access to all the Confluent Cloud services with simplified experience and billing
- 28. Connect
- 31. ksqlDB
- 32. What’s stream processing good for? Materialized cache Build and serve incrementally updated stateful views of your data. 32 Streaming ETL pipeline Manipulate in-flight events to connect arbitrary sources and sinks. Event-driven microservice Trigger changes based on observed patterns of events in a stream.
- 33. 3 modalities of stream processing with Confluent Kafka clients 33 Kafka Streams ksqlDB ConsumerRecords<String, String> records = consumer.poll(100); Map<String, Integer> counts = new DefaultMap<String, Integer>(); for (ConsumerRecord<String, Integer> record : records) { String key = record.key(); int c = counts.get(key) c += record.value() counts.put(key, c) } for (Map.Entry<String, Integer> entry : counts.entrySet()) { int stateCount; int attempts; while (attempts++ < MAX_RETRIES) { try { stateCount = stateStore.getValue(entry.getKey()) stateStore.setValue(entry.getKey(), entry.getValue() + stateCount) break; } catch (StateStoreException e) { RetryUtils.backoff(attempts); } } } builder .stream("input-stream", Consumed.with(Serdes.String(), Serdes.String())) .groupBy((key, value) -> value) .count() .toStream() .to("counts", Produced.with(Serdes.String(), Serdes.Long())); SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
- 34. Using external processing systems leads to complicated architectures DB CONNECTOR APP APP DB STREAM PROCESSING APPDB CONNECTOR CONNECTOR
- 35. We can put it back together in a simpler way DB APP APP DB APP PULL PUSH CONNECTORS STREAM PROCESSING STATE STORES ksqlDB
- 36. Build a complete streaming app with one mental model in SQL Serve lookups against materialized views Create materialized views Perform continuous transformations Capture data CREATE STREAM purchases AS SELECT viewtime, userid,pageid, TIMESTAMPTOSTRING(viewtime, 'yyyy-MM-dd') FROM pageviews; CREATE TABLE orders_by_country AS SELECT country, COUNT(*) AS order_count, SUM(order_total) AS order_total FROM purchases WINDOW TUMBLING (SIZE 5 MINUTES) LEFT JOIN user_profiles ON purchases.customer_id = user_profiles.customer_id GROUP BY country EMIT CHANGES; SELECT * FROM orders_by_country WHERE country='usa'; CREATE SOURCE CONNECTOR jdbcConnector WITH ( ‘connector.class’ = '...JdbcSourceConnector', ‘connection.url’ = '...', …);
- 37. Summary
- 38. Flexible: Confluent Cloud has a Kafka cluster type for every need DedicatedBasic Standard Elastic scaling up to 100 MBps, pay only for what you stream Elastic scaling up to 100 MBps, multi-zone HA, enterprise-grade SLA, private networking coming soon GBps scale, dedicated compute and storage, HIPAA compliance, multiple private networking options Developers and price sensitive production use cases that don’t require advanced features Mainstream production workloads Large-scale production workloads (very high throughput, high security, high SLA, etc) Target use cases Features Entry level & low cost Enterprise features with low prices Single tenant w/ GBps scale
- 40. Try it on the Azure Marketplace for free! Get up to $200 off your first 3 Confluent Cloud months Thank You!