Confluent Platform 5.4 + Apache Kafka 2.4 Overview (RBAC, Tiered Storage, Multi-Region Clusters, Audit Logs, and more)
0 likes2,033 views
The document provides an overview of Confluent Platform 5.4, highlighting features such as role-based access control, multi-region clustering, improved security measures, and enhanced management tools. It emphasizes the benefits of these features for enterprise environments, including better data quality, compliance, and disaster recovery capabilities. Additionally, it outlines various deployment options and the support for SQL-based stream processing through ksqlDB.
Confluent Platform 5.4 + Apache Kafka 2.4 Overview (RBAC, Tiered Storage, Multi-Region Clusters, Audit Logs, and more)
- 1. Confluent Platform 5.4 - RBAC, Multi-Region Clusters and more March 25th 2020 Kai Waehner Technology Evangelist kai.waehner@confluent.io LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. Ways to Deploy Confluent Platform The enterprise distribution of Apache Kafka VM Deploy on any platform on-prem or cloud Self Managed Software Fully Managed Software Confluent Cloud Apache Kafka re-engineered for the Cloud Available on the leading public clouds Also via MarketplaceAlso via Marketplace
- 3. Confluent Platform Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterpris e Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer
- 4. Rapid Pace of Innovation to Enable Enterprises January 2020 CP 5.4 (based on AK 2.4) Security ● Role-Based Access Control ● Structured Audit Logs Resilience ● Multi-Region Clusters Data Compatibility ● Schema Validation Management & Monitoring ● Control Center ○ RBAC management ○ Replicator monitoring Performance & Elasticity ● Tiered Storage (preview) Stream Processing ● ksqlDB features (preview) April 2019 CP 5.2 (based on AK 2.2) Developers ● Free single-broker developer license ● librdkafka and clients 1.0 KSQL ● New query expressions ● GUI enhancements Replicator ● Schema migration to CCloud Control Center ● Dynamix broker configuration ● Schema Registry management ● Multi-cluster Connect & KSQL ● Enhanced scalability July 2018 CP 5.0 (based on AK 2.0) Security ● AD/LDAP Authorizer Replicator ● Automatic offset translation Control Center ● Consumer lag ● View broker configuration ● View topics ● KSQL editor Ecosystem ● MQTT Proxy July 2019 CP 5.3 (based on AK 2.3) Security ● Role-Based Access Control (preview) ● Secret Protection DevOps automation ● Kubernetes Operator ● Ansible Playbooks Management & Monitoring ● Control Center redesigned user interface ● New CLI
- 6. Enterprise Grade Security • Architecting with security is a design priority • Avoiding unnecessary complexity is key • As usage of event streaming spreads, native tools (e.g. Kafka Access Control Lists) for managing authorization can become complex • Problem is exacerbated when failing to standardize security across the platform 6 Why you need better authorization?
- 7. Role-Based Access Control Provides platform-wide security with fine-tuned granularity • Granular control of access permissions, including: • Clusters, topics, consumer groups, connectors • Efficient management at large scale • Delegate authorization management to true resource owners • Platform-wide standardization • Enforced via GUI, CLI and APIs • Enforced across all CP components: Connect, KSQL, Schema Registry, REST Proxy, Control Center and MQTT Proxy Users/ Groups Roles Resource scoping CLI GUI API Role Binding RBAC authorization 7
- 8. Enterprise Grade Security • Lack of visibility into actions taken by users/applications • Difficult to perform forensics to detect anomalies and identify bad actors • Failure to comply with regulatory requirements 8 Why you need better visibility?
- 9. Structured Audit Logs Enable security traceability and regulatory compliance • Detection of abnormal behavior and potential security threats • Capture authorization logs in a set of dedicated Kafka topics • Process and analyze with KSQL, or offload to external systems (e.g. Splunk, S3) • Industry Standardization • Uses CloudEvents specification to define the syntax of the logs Event Description Category Capture Default Authorize An RBAC authorization is being requested. MANAGEMENT Yes CreateTopics A topic is being created. MANAGEMENT Yes Produce A Kafka producer is writing a batch of records to a topic. PRODUCE No FetchConsumer A Kafka consumer is reading a batch of records from a topic. CONSUME No LeaderAndIsr Controller is sending leader and ISR state to a broker. INTERBROKER No Sample Audit Logs 9
- 11. Global Resilience • Modern companies have high expectations for durability, availability, and latency • Replication based on Kafka Connect (e.g. Replicator or MirrorMaker 2) come with operational complexity and require downtime • Stretch cluster architectures historically came with a tradeoff: availability vs. performance 11 Why you need better disaster recovery?
- 12. Multi-Region Clusters Change the game for disaster recovery for Kafka • Zero downtime and zero data loss for critical Kafka Topics • Automated client failover • Streamlined DR operations • Leverages Kafka’s internal replication • No separate Connect clusters • Single multi-region cluster with high write throughput • Asynchronous replication using “Observer” replicas • Low bandwidth costs and high read throughput • Remote consumers read data locally, directly from Observers Broker 1 Broker 2 Broker 3 ZK1 Broker 4 Broker 5 Broker 6 Broker 1 Broker 2 ZK2 Client D Client F Client G Failover site ZK3 Broker 3 Broker 4 Broker 5 Broker 6 Client A Client B us-central-1 Client A Client B automated client failover Observer replicas us-west-1 us-east-1 Site failure! “tie-breaker” datacenter Single Kafka Cluster 12
- 14. Data Compatibility • Confluent Schema Registry increase data compatibility through client- level “agreements”, but Kafka is unaware • No programmatic way of enforcing that producers talk to Schema Registry before publishing messages to Kafka • Leads to risk and uncertainty regarding data quality for large organizations 14 Why you need enhanced validation of of data quality?
- 15. Schema Validation Provides a centralized way of controlling data compatibility • Certainty and piece of mind at scale regarding data quality • Automated broker-side schema validation and enforcement • Direct interface from the broker to Confluent Schema Registry • Granular control over schema validation • Enabled at the topic level Producer Broker Schema Registry 1. Invalid schema 2. Error message confluent.value.schema.validation=true 15
- 16. GUI -Driven Management & Monitoring
- 17. GUI Driven Management & Monitoring • Control Center is rapidly becoming the de facto user interface for many Confluent Platform users • We must ensure that Control Center can manage and monitor Confluent Platform comprehensively • Need for a wide variety of use cases and supported scale 17 Why these improvements to Control Center?
- 18. GUI-driven mgmt for new CP 5.4 features • Role-Based Access Control • View own permissions, and manage subordinate role bindings • Multi-Region Clusters • Track Observer replica placement in each topic view • Schema Validation • Enable at the topic level when creating or editing topics RBAC management 18
- 19. Confluent Replicator integration • Simplified monitoring for multi-site replication with the GUI • Track key metrics such as throughput and lag Replicator monitoring 19
- 20. New aggregate views • Simplified monitoring and troubleshooting for Kafka clusters • Cluster Overview: shows overall status of the Kafka cluster, including brokers, replicas, partitions and topics • Metrics Dashboard: aggregates all Kafka cluster metrics into a single page Cluster Overview Metrics Dashboard 20
- 21. Dynamic Performance & Elasticity
- 22. Dynamic Performance & Elasticity • As event streaming spreads, the platform is required to store larger amounts of data for longer periods of time • Kafka’s tight coupling between compute and storage leads to difficulty to scale the platform • Longer data retention leads to high storage costs 22 Why you need enhanced scalability and data efficiency ?
- 23. Tiered Storage (preview) Enable Kafka with infinite retention cost- effectively • Infinite retention • Older data is offloaded to inexpensive object storage, accessible at any time • Reduced storage costs • Storage limitations, like capacity and duration, are effectively uncapped • Elastic scalability • “Lighter” Kafka brokers enable instantaneous load balancing when scaling up Broker Compute Storage Clients Transactions, auth, quota enforcement, compaction, ... Local Remote Object Storage 23
- 24. Confluent Server Enables enterprise features Required to enable: ● Operator ● RBAC ● Structured Audit Logs ● Multi-Region Clusters ● Schema Validation ● Tiered Storage (preview) Optional software package ● Deploy CP with Confluent Server or Apache Kafka ● In-place migration between Confluent Server and Kafka Apache Kafka enterprise capabilities Confluent Server Confluent Platform KSQL Schema Registry REST Proxy Control Center Replicator MQTT Proxy 24
- 26. Kafka producer/ consumer Kafka Streams ksqlDB The 3 stream processing modalities with Confluent
- 27. ConsumerRecords<String, String> records = consumer.poll(100); Map<String, Integer> counts = new DefaultMap<String, Integer>(); for (ConsumerRecord<String, Integer> record : records) { String key = record.key(); int c = counts.get(key) c += record.value() counts.put(key, c) } for (Map.Entry<String, Integer> entry : counts.entrySet()) { int stateCount; int attempts; while (attempts++ < MAX_RETRIES) { try { stateCount = stateStore.getValue(entry.getKey()) stateStore.setValue(entry.getKey(), entry.getValue() + stateCount) break; } catch (StateStoreException e) { RetryUtils.backoff(attempts); } } } The 3 stream processing modalities differ in flexibility and ease of use Kafka producer/consumer Kafka Streams ksqlDB builder .stream("input-stream", Consumed.with(Serdes.String(), Serdes.String())) .groupBy((key, value) -> value) .count() .toStream() .to("counts", Produced.with(Serdes.String(), Serdes.Long())); SELECT x, count(*) FROM stream GROUP BY x EMIT CHANGES;
- 28. Using external processing systems leads to complicated architectures DB CONNECTOR CONNECTOR APP APP DB STREAM PROCESSING CONNECTOR APPDB
- 29. We can put it back together in a simpler way DB APP APP DB APP PULL PUSH CONNECTORS STREAM PROCESSING STATE STORES ksqlDB
- 30. Connect integration and pull queries enable end-to-end streaming in just a few SQL statements Serve lookups against materialized views Create materialized views Perform continuous transformations Capture data CREATE STREAM purchases AS SELECT viewtime, userid,pageid, TIMESTAMPTOSTRING(viewtime, 'yyyy-MM-dd HH:mm:ss.SSS') FROM pageviews; CREATE TABLE orders_by_country AS SELECT country, COUNT(*) AS order_count, SUM(order_total) AS order_total FROM purchases WINDOW TUMBLING (SIZE 5 MINUTES) LEFT JOIN user_profiles ON purchases.customer_id = user_profiles.customer_id GROUP BY country EMIT CHANGES; SELECT * FROM orders_by_country WHERE country='usa'; CREATE SOURCE CONNECTOR jdbcConnector WITH ( ‘connector.class’ = '...JdbcSourceConnector', ‘connection.url’ = '...', …);
- 32. Confluent Platform and Confluent Cloud are always built on the latest Version of Apache Kafka If you want to learn what’s included in Apache Kafka 2.4, we have resources available for you: • Technical Blog: https://www.confluent.io/blog/apac he-kafka-2-4-latest-version-updates • Overview Video: https://youtu.be/Ipzc--mbvzg 32 Apache Kafka 2.4