Beyond the brokers - Un tour de l'écosystème Kafka
- 1. 1 1 Beyond the brokers A tour of the Kafka Ecosystem Florent Ramière TAM / System Engineer florent@confluent.io
- 2. 2 Massive volumes of new data generated every day Mobile Cloud Microservices Internet of Things Machine Learning Distributed across apps, devices, datacenters, clouds Structured, unstructured polymorphic What
- 3. 3 With
- 4. 4 How
- 5. 5 Store & ETL Process Publish & Subscribe In short
- 6. 6 From a simple idea
- 7. 7 From a simple idea
- 8. 8 with great properties ! • Scalability • Retention • Durability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • …
- 9. 9 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Investment & Time Solve A Critical Need Maturity model
- 10. 10 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 11. 11 Set up secure Kafka & build your first app Understand streaming Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Infrastructure & apps across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streaming Self-service on shared Kafka Infrastructure & applications across LOBs Monitor & manage a mission-critical solution Set up secure Kafka & build your first app Understand streamingUnderstand streaming Pre-streamingValue Stream Everything 05Break Silos 04 03 Go To Production 02 Learn Kafka 01 Solve A Critical Need Maturity model
- 12. 12 12 Platform
- 13. 13 … spawned a full platform Apache Kafka® Core | Connect API | Streams API Stream Processing & Compatibility KSQL | Schema Registry Operations Replicator | Auto Data Balancer | Connectors | MQTT Proxy | Operator Database Changes Log Events IoT Data Web Events other events Hadoop Database Data Warehouse CRM other DATA INTEGRATION Transformations Custom Apps Analytics Monitoring other REAL-TIME APPLICATIONS OPEN SOURCE FEATURES COMMERCIAL FEATURES Datacenter Public Cloud Confluent Cloud CONFLUENT PLATFORM Administration & Monitoring Control Center | Security Connectivity Clients | Connectors | REST Proxy CONFLUENT FULLY-MANAGEDCUSTOMER SELF-MANAGED
- 14. 14 14 ETL
- 15. 15 Apache Kafka Connect API: Import and Export Data In & Out of Kafka JDBC Mongo MySQL Elastic Cassandra HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center
- 16. 16 Connectors: Connect Kafka Easily with Data Sources and Sinks Databases Datastore/File Store Analytics Applications / Other
- 17. 17 Kafka Connect API, Part of the Apache Kafka™ Project Connect any source to any target system Integrated • 100% compatible with Kafka v0.9 and higher • Integrated with Confluent’s Schema Registry • Easy to manage with Confluent Control Center Flexible • 40+ open source connectors available • Easy to develop additional connectors • Flexible support for data types and formats Compatible • Maintains critical metadata • Preserves schema information • Supports schema evolution Reliable • Automated failover • Exactly-once guarantees • Balances workload between nodes
- 18. 18 Confluent Hub - The Kafka App Store
- 20. 20 Clients: Communicate with Kafka in a Broad Variety of Languages Apache Kafka Confluent Platform Community Supported Proxy http/REST stdin/stdout Confluent Platform Clients developed and fully supported by Confluent
- 21. 21 REST Proxy: Talking to Non-native Kafka Apps and Outside the Firewall REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster
- 22. 22 22 IOT
- 23. 23 MQTT Proxy: Streamline IoT Data Integration with Kafka Connect all IoT data sources with the streaming platform - leverages all of your infrastructure investments Reduce operational cost and complexity by eliminating third party MQTT brokers and their intermediate storage and lag Ensure IoT data delivery at all QoS levels (QoS0, QoS1 and QoS2) of the MQTT protocol Kafka Broker Kafka Broker Kafka Broker MQTT ProxyGatewaysDevices MQTT MQTT
- 24. 24 ? Frictionless MQTT Connectivity with Confluent Platform Kafka BrokerKafka BrokerKafka BrokerDevicesDevicesDevicesGateways MQTT Broker Connect w/ MQTT connector Connect w/ MQTT connectorMQTT DevicesDevicesDevicesDevices MQTT DevicesDevicesDevicesDevices Kafka BrokerKafka BrokerKafka Broker MQTT ProxyMQTT DevicesDevicesDevicesGateways DevicesDevicesDevicesDevices MQTT DevicesDevicesDevicesDevices Approach 1: Integrate 3rd Party MQTT Broker(s) with Kafka Connect : Approach 2: Integrate MQTT clients directly via MQTT Proxy (CP 5.x and later) :
- 25. 25 25 Processing
- 26. 26 Stream Processing by Analogy Kafka Cluster Connect API Stream Processing Connect API $ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt
- 27. 27 • subscribe() • poll() • send() • flush() Consumer, Producer Flexibility Simplicity Trade offs
- 28. 28 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams Flexibility Simplicity Trade offs
- 33. 33 Things Kafka Streams Does Runs everywhere Clustering done for you Exactly-once processing Event-time processing Integrated database Joins, windowing, aggregation S/M/L/XL/XXL/XXXL sizes
- 34. 34 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams Flexibility Simplicity Trade offs
- 35. 35 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity Trade offs
- 36. 36 KSQL for Data Exploration SELECT status, bytes FROM clickstream WHERE user_agent = 'Mozilla/5.0 (compatible; MSIE 6.0)';
- 37. 37 KSQL for Streaming ETL CREATE STREAM vip_actions AS SELECT userid, page, action FROM clickstream c LEFT JOIN users u ON c.userid = u.user_id WHERE u.level = 'Platinum';
- 38. 38 Nested Types SELECT eventid, address.city FROM users WHERE address.state = 'CA';
- 39. 39 User Defined Functions (UDF) SELECT eventid, anomaly(sensorinput) FROM sensor @Udf(description = "apply analytic model to sensor input") public String anomaly(String sensorinput){ return your_logic; }
- 40. 40 KSQL for Anomaly Detection CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
- 41. 41 KSQL: Enable Stream Processing using SQL-like Semantics Example Use Cases • Streaming ETL • Anomaly detection • Event monitoring Leverage Kafka Streams API without any coding required KSQL server Engine (runs queries) REST API CLIClients Confluent Control Center GUI Kafka Cluster Use any programming language Connect via CLI or Control Center user interface
- 42. 42 KSQL is really Kafka Stream ? ... yes!
- 43. 43 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity Trade offs
- 44. 44 Lowering the Bar to Enter the World of Streaming Kafka User Population CodingSophistication Core Java developers Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts streams
- 45. 45 45 Schema
- 46. 46 The Challenge of Data Compatibility at Scale : implicit → explicit ! App 1 App 2 App 3 Many sources without a policy causes mayhem in a centralized data pipeline Ensuring downstream systems can use the data is key to an operational stream pipeline Example: Date formats Even within a single application, different formats can be presented Incompatibly formatted message
- 47. 47 Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry
- 48. 48 48 Deployment
- 49. 49 Which one do you prefer ? • Zip • Yum/apt • Ansible • Docker • DC/OS • Helm-charts • Confluent Operator • ... Cloud!
- 50. 50 Operator: Achieve End to End Automation on Kubernetes Confluent Platform Docker Images Public Cloud On-Premises Pivotal Mesosphere Red HatAWS Azure GCP Confluent Operator operationalizes years of experience delivering a fully-managed service - Confluent Cloud - on the leading public clouds Confluent Cloud Docker Images Confluent Operator Accelerate time to value with automated zero-touch provisioning Reduce OpEx and boost DevOps agility with rolling updates, elastic scaling and auto data balancing Increase resiliency via SLA monitoring through Control Center or Prometheus
- 51. 51 Amazon Elastic Container Service (EKS) Google Kubernetes Engine (GKE) Azure Kubernetes Service (AKS) KUBERNETES DISTRIBUTIONS SERVICE PROVIDERS
- 52. 52 52 Tools
- 53. 53 Auto Data Balancer: Achieve Enterprise-level Performance for Kafka Before After Rebalance Dynamically move partitions to optimize resource utilization and reliability Enable elastic scaling by easily adding and removing nodes from your Kafka cluster ADB traffic is throttled upon data transfers to ensure network bandwidth
- 54. 54 Replicator: Stretch Kafka Across Data Centers and Public Cloud Protect business-critical data and metadata by replicating down to topic-level configurations Minimize recovery time objectives (RTO) through automated failover and switchback Meet recovery point objectives (RPO) running more workers to increase replication throughput Bridge your data center to the cloud with Confluent Cloud
- 55. 55 55 Monitoring
- 56. 56 System Health Are all brokers and topics available? How much data is being processed? What can be tuned to improve performance? End-to-End SLA Monitoring Does Kafka process all events <15 seconds? Is the 8am report missing data? Are there duplicate events?
- 58. 58 Confluent Control Center– Cluster Health & Administration Cluster health dashboard • Monitor the health of your Kafka clusters and get alerts if any problems occur • Measure system load, performance, and operations • View aggregate statistics or drill down by broker or topic Cluster administration • Monitor topic configurations
- 59. 59 Operate More Secure, Reliable and Performant Apache Kafka ● Broker configuration view → see config across multiples Kafka clusters or check values for specific brokers ● Consumer lag → view how consumers are performing based on offset, spot potential issues and take proactive steps to keep performance high ● Feature access controls → control customer access to topic inspection, schemas, and KSQL For Operators
- 60. 60 Build More Powerful Streaming Applications ● Topic inspection → gain insight into the actual data in Kafka topics ● Schema registry integration → view older and current schema versions in a git-like UI ● KSQL GUI → create streams and tables from topics, experiment with transient queries, and run persistent queries to filter and enrich data For Developers
- 61. 61 View consumer-partition lag across topics for a consumer group Alert on max consumer group lag across all topics Consumer Lag Monitoring 61
- 62. 62 Make stream processing more accessible Build stream processing IP in CE Manage streams & tables Run KSQL (transient & persistent) View persistent queries KSQL UI 62
- 63. 63 63 Resources
- 65. 65 Optimizing Your Apache Kafka® Deployment https://www.confluent.io/white-paper/optimizing-your-apache-kafka-deployment/
- 66. 66 Resources - Confluent Enterprise Reference Architecture https://www.confluent.io/whitepaper/confluent-enterprise-reference-architecture/
- 67. 67 Large Cluster Reference Architecture – 22 software nodes - 19 hosts 3 nodes 4 nodes 5 nodes 4 nodes 2 nodes 2 nodes 1 node
- 68. 68 68 Community
- 69. 69 Resources – Community Slack and Mailing List https://slackpass.io/confluentcommunity https://groups.google.com/forum/#!forum/confluent-platform
- 71. 71 Confluent Platform Demo : cp-demo https://github.com/confluentinc/cp-demo With security inside!
- 72. 72 Examples Examples Examples ! https://github.com/confluentinc/examples
- 75. 75 75 Take Away
- 76. 76 Kafka Provides a Central Nervous System for the Modern Digital Enterprise Enabling companies to respond accurately and in real time to business events
- 77. 77 77 Let’s talk ! ☺ florent@confluent.io