Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
1 like328 views
The document outlines the features and functionalities of Apache Kafka and the Confluent platform, emphasizing its ability to manage large volumes of data from various sources through robust architectures. Key components discussed include Kafka's scalability, resilience, and the integration of its Connect API for data import/export, stream processing via ksql, and monitoring capabilities. Additionally, it highlights deployment options, automation through Kubernetes, and the importance of data compatibility facilitated by the schema registry.
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
- 1. 1 Au-delà des brokers, un tour de l’environement Kafka Florent Ramière @framiere Technical Account Manager/SE Confluent PARIS - 11 OCTOBRE 2018
- 3. 3 Massive volumes of new data generated every day Mobile Cloud Microservices Internet of Things Machine Learning Distributed across apps, devices, datacenters, clouds Structured, unstructured polymorphic What
- 4. 4 With
- 5. 5 How
- 6. 6 Store & ETL Process Publish & Subscribe In short
- 7. 7 From a simple idea
- 8. 8 From a simple idea
- 9. 9 … with great properties • Scalability • Replication • Security • Resiliency • Throughput • Ordering • Exactly Once Semantic • Transaction • Idempotency • Immutability • Performance • …
- 10. 10 10 Platform
- 11. 11 … spawned a full platform Apache Kafka® Core | Connect API | Streams API Stream Processing & Compatibility KSQL | Schema Registry Operations Replicator | Auto Data Balancer | Connectors | MQTT Proxy | Operator Database Changes Log Events IoT Data Web Events other events Hadoop Database Data Warehouse CRM other DATA INTEGRATION Transformations Custom Apps Analytics Monitoring other REAL-TIME APPLICATIONS OPEN SOURCE FEATURES COMMERCIAL FEATURES Datacenter Public Cloud Confluent Cloud CONFLUENT PLATFORM Administration & Monitoring Control Center | Security Connectivity Clients | Connectors | REST Proxy CONFLUENT FULLY-MANAGEDCUSTOMER SELF-MANAGED
- 12. 12 12 ETL
- 13. 13 Apache Kafka Connect API: Import and Export Data In & Out of Kafka JDBC Mongo MySQL Elastic Cassandra HDFS Kafka Connect API Kafka Pipeline Connector Connector Connector Connector Connector Connector Sources Sinks Fault tolerant Manage hundreds of data sources and sinks Preserves data schema Integrated within Confluent Control Center
- 14. 14 Connectors: Connect Kafka Easily with Data Sources and Sinks Databases Datastore/File Store Analytics Applications / Other Orange Logo denotes Connectors developed and fully supported by Confluent
- 15. 15 Kafka Connect API, Part of the Apache Kafka™ Project Connect any source to any target system Integrated • 100% compatible with Kafka v0.9 and higher • Integrated with Confluent’s Schema Registry • Easy to manage with Confluent Control Center Flexible • 40+ open source connectors available • Easy to develop additional connectors • Flexible support for data types and formats Compatible • Maintains critical metadata • Preserves schema information • Supports schema evolution Reliable • Automated failover • Exactly-once guarantees • Balances workload between nodes
- 16. 16 Confluent Hub - The Kafka App Store
- 18. 18 Clients: Communicate with Kafka in a Broad Variety of Languages Apache Kafka Confluent Platform Community Supported Proxy http/REST stdin/stdout Confluent Platform Clients developed and fully supported by Confluent
- 19. 19 REST Proxy: Talking to Non-native Kafka Apps and Outside the Firewall REST Proxy Non-Java Applications Native Kafka Java Applications Schema Registry REST / HTTP Simplifies administrative actions Simplifies message creation and consumption Provides a RESTful interface to a Kafka cluster
- 20. 20 20 IOT
- 21. 21 MQTT Proxy: Streamline IoT Data Integration with Kafka Connect all IoT data sources with the streaming platform - leverages all of your infrastructure investments Reduce operational cost and complexity by eliminating third party MQTT brokers and their intermediate storage and lag Ensure IoT data delivery at all QoS levels (QoS0, QoS1 and QoS2) of the MQTT protocol Kafka Broker Kafka Broker Kafka Broker MQTT ProxyGatewaysDevices MQTT MQTT
- 22. 22 ? Frictionless MQTT Connectivity with Confluent Platform Kafka BrokerKafka BrokerKafka BrokerDevicesDevicesDevicesGateways MQTT Broker Connect w/ MQTT connector Connect w/ MQTT connectorMQTT DevicesDevicesDevicesDevices MQTT DevicesDevicesDevicesDevices MQTT Kafka BrokerKafka BrokerKafka Broker MQTT ProxyMQTT DevicesDevicesDevicesGateways DevicesDevicesDevicesDevices MQTT DevicesDevicesDevicesDevices MQTT Approach 1: Integrate 3rd Party MQTT Broker(s) with Kafka Connect : Approach 2: Integrate MQTT clients directly via MQTT Proxy (CP 5.x and later) :
- 23. 23 23 Processing
- 24. 24 Stream Processing by Analogy Kafka Cluster Connect API Stream Processing Connect API $ cat < in.txt | grep "ksql" | tr a-z A-Z > out.txt
- 25. 25 • subscribe() • poll() • send() • flush() Consumer, Producer • mapValues() • filter() • punctuate() Kafka Streams • Select…from… • Join…where… • Group by.. KSQL Flexibility Simplicity Trade offs
- 26. 26 KSQL: Enable Stream Processing using SQL-like Semantics Example Use Cases • Streaming ETL • Anomaly detection • Event monitoring Leverage Kafka Streams API without any coding required KSQL server Engine (runs queries) REST API CLIClients Confluent Control Center GUI Kafka Cluster Use any programming language Connect via CLI or Control Center user interface
- 27. 27 CREATE TABLE possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 SECONDS) GROUP BY card_number HAVING count(*) > 3;
- 28. 28 28 Schema
- 29. 29 The Challenge of Data Compatibility at Scale App 1 App 2 App 3 Many sources without a policy causes mayhem in a centralized data pipeline Ensuring downstream systems can use the data is key to an operational stream pipeline Example: Date formats Even within a single application, different formats can be presented Incompatibly formatted message
- 30. 30 Schema Registry: Make Data Backwards Compatible and Future-Proof ● Define the expected fields for each Kafka topic ● Automatically handle schema changes (e.g. new fields) ● Prevent backwards incompatible changes ● Support multi-data center environments Elastic Cassandra HDFS Example Consumers Serializer App 1 Serializer App 2 ! Kafka Topic! Schema Registry
- 31. 31 31 Deployment
- 32. 32 Multiple options! • Zip • Yum/apt • Ansible • Docker • DC/OS • Helm-charts • Confluent Operator • ... Cloud!
- 33. 33 Operator: Achieve End to End Automation on Kubernetes Confluent Platform Docker Images Public Cloud On-Premises Pivotal Mesosphere Red HatAWS Azure GCP Confluent Operator operationalizes years of experience delivering a fully-managed service - Confluent Cloud - on the leading public clouds Confluent Cloud Docker Images Confluent Operator Accelerate time to value with automated zero-touch provisioning Reduce OpEx and boost DevOps agility with rolling updates, elastic scaling and auto data balancing Increase resiliency via SLA monitoring through Control Center or Prometheus
- 34. 34 34 Tools
- 35. 35 Auto Data Balancer: Achieve Enterprise-level Performance for Kafka Before After Rebalance Dynamically move partitions to optimize resource utilization and reliability Enable elastic scaling by easily adding and removing nodes from your Kafka cluster ADB traffic is throttled upon data transfers to ensure network bandwidth
- 36. 36 Replicator: Stretch Kafka Across Data Centers and Public Cloud Protect business-critical data and metadata by replicating down to topic-level configurations Minimize recovery time objectives (RTO) through automated failover and switchback Meet recovery point objectives (RPO) running more workers to increase replication throughput Bridge your data center to the cloud with Confluent Cloud
- 37. 37 Deploy Confluent Platform on K8s via a Growing Partner Ecosystem
- 38. 38 38 Monitoring
- 39. 39 Confluent Control Center– Cluster Health & Administration Cluster health dashboard • Monitor the health of your Kafka clusters and get alerts if any problems occur • Measure system load, performance, and operations • View aggregate statistics or drill down by broker or topic Cluster administration • Monitor topic configurations
- 40. 40 Operate More Secure, Reliable and Performant Apache Kafka ● Broker configuration view → see config across multiples Kafka clusters or check values for specific brokers ● Consumer lag → view how consumers are performing based on offset, spot potential issues and take proactive steps to keep performance high ● Feature access controls → control customer access to topic inspection, schemas, and KSQL For Operators Control Center enhancements in Confluent Platform 5.0
- 41. 41 Build More Powerful Streaming Applications ● Topic inspection → gain insight into the actual data in Kafka topics ● Schema registry integration → view older and current schema versions in a git-like UI ● KSQL GUI → create streams and tables from topics, experiment with transient queries, and run persistent queries to filter and enrich data Control Center enhancements in Confluent Platform 5.0 For Developers
- 42. 42 View consumer-partition lag across topics for a consumer group Alert on max consumer group lag across all topics Consumer Lag Monitoring 42 Control Center enhancements in Confluent Platform 5.0
- 43. 43 Make stream processing more accessible Build stream processing IP in CE Manage streams & tables Run KSQL (transient & persistent) View persistent queries KSQL UI 43 Control Center enhancements in Confluent Platform 5.0
- 45. 45 45
- 47. 47 Resources - Confluent Cloud Datasheet https://www.confluent.io/wp-content/uploads/confluent-cloud-datasheet.pdf
- 48. 48 Resources - Confluent Enterprise Reference Architecture https://www.confluent.io/whitepaper/confluent-enterprise-reference-architecture/
- 49. 49 Optimizing Your Apache Kafka® Deployment https://www.confluent.io/white-paper/optimizing-your-apache-kafka-deployment/
- 50. 50 Small Cluster Reference Architecture – 19 software nodes – 8 Hosts 10 nodes 6 nodes 2 nodes 1 node
- 51. 51 Large Cluster Reference Architecture – 22 software nodes - 19 hosts 3 nodes 4 nodes 5 nodes 4 nodes 2 nodes 2 nodes 1 node
- 52. 52 52 Community
- 53. 53 Resources – Community Slack and Mailing List https://slackpass.io/confluentcommunity https://groups.google.com/forum/#!forum/confluent-platform
- 58. 58 PARIS - 11 OCTOBRE 2018