Team Collaboration in Kafka Clusters With Maria Berinde-Tampanariu | Current 2022
0 likes517 views
The document discusses the collaborative capabilities and management of Kafka clusters for different teams, emphasizing authentication, authorization, and naming conventions. It details various authentication mechanisms, access control methods including ACL and RBAC, as well as platform limits and client quotas. The importance of automation, monitoring, and chargeback processes for effective Kafka management is also highlighted.
![Access Control List (ACL)
• general format:
"Principal P is [Allowed/Denied] Operation O From Host H On Resource R"
• wildcard & prefix matching supported
10
Principal P
based on standard
authorizer
(wildcard)
is [Allowed
/ Denied]
Operation O From Host H On Resource R
(wildcard & prefix)
Apache
Kafka®
individual principals
“Deny”
always
trumps
“Allow”.
supported
operations are
based on resource
(see docs)
supported
Cluster
Delegation Token
Group
Topic
Transactional ID
Confluent
Platform
individual & group
principals
Confluent
Cloud
user & service accounts
not
supported
Cluster
Consumer Group
Topic
Transactional ID](https://image.slidesharecdn.com/mariaberinde-tampanariubr02320221003final-221021141748-056de83d/85/Team-Collaboration-in-Kafka-Clusters-With-Maria-Berinde-Tampanariu-Current-2022-10-320.jpg)
Team Collaboration in Kafka Clusters With Maria Berinde-Tampanariu | Current 2022
- 1. Team Collaboration in Kafka Clusters Maria Berinde-Tâmpănariu Advisory Solutions Engineer 4th of October 2022
- 2. Question Can different teams collaborate efficiently and work independently in your Kafka cluster(s)?
- 3. Journey 3 My first Kafka cluster ● a foreseeable amount of applications ● the core team with full access Central Nervous System ● many different types of clients ● many users with different access levels The ability to work without getting in each other’s way. ➔ scalable & repeatable actions ➔ predictability ➔ self-service capabilities ➔ isolation ➔ manageability
- 4. Authentication Authorization Naming Conventions Automation TEAM COLLABORATION Platform Limits Chargeback Monitoring Client Quotas
- 7. Client Authentication • process of establishing the client identity and verifying client & server authenticity • authenticated identity throughout lifetime of connection • KafkaPrincipal used to represent client identity (e.g. Username: maria) • principal used to: - grant access to resources - allocate quotas - log details • different authentication mechanisms 7
- 8. Authentication Methods Confluent Cloud 8 API Keys OAuth Single Sign On • Cloud keys • resource specific keys - Kafka - Schema Registry - ksqlDB • all keys owned by an account • key rotation • delegated authentication • JSON Web Token (JWT) • OpenID Connect (OIDC) • identity provider & identity pools • SAML based Identity Provider (IdP) • enabled at Confluent Cloud organization level • SSO users vs. local users Confluent Cloud is a fully-managed Apache Kafka service available on all three major clouds. • user & service accounts
- 10. Access Control List (ACL) • general format: "Principal P is [Allowed/Denied] Operation O From Host H On Resource R" • wildcard & prefix matching supported 10 Principal P based on standard authorizer (wildcard) is [Allowed / Denied] Operation O From Host H On Resource R (wildcard & prefix) Apache Kafka® individual principals “Deny” always trumps “Allow”. supported operations are based on resource (see docs) supported Cluster Delegation Token Group Topic Transactional ID Confluent Platform individual & group principals Confluent Cloud user & service accounts not supported Cluster Consumer Group Topic Transactional ID
- 11. Authorizer • customizable server plugin • authorize an operation based on the principal and the resource being accessed 11 Confluent Cloud . • subset of Kafka Access Control Lists (ACL) • predefined role-based access control (RBAC) roles • ACL & RBAC can be used together • AclAuthorizer (since v5.4.0) • SimpleAclAuthorizer (before v5.4.0) • Confluent Server Authorizer with LDAP group-based & role-based access control (RBAC) ‘ • Access Control Lists (ACL) stored on Zookeeper (ZK) or centrally on Metadata Service (MDS) Confluent Platform Apache Kafka® • pluggable Authorizer • out-of-box implementation • default authorizer: AclAuthorizer ( > v2.4) SimpleAclAuthorizer (< v2.4) StandardAuthorizer (KRaft) • Access Control Lists (ACL) stored on Zookeeper (ZK) or in metadata topic
- 12. Role-based Access Control (RBAC) • serves as an additional authorization layer on top of ACLs • predefined roles & role-bindings • Metadata Service used to configure and manage RBAC • only “Allow” rules (“Deny” not supported) • benefits: + Manage security access across the platform (Kafka, ksqlDB, Connect, Schema Registry, Confluent Control Center) + delegation of permission management is possible (ResourceOwner role) + centrally manage multiple clusters 12
- 13. RBAC on Confluent Cloud CLI GUI API Org Admin Env Admin Env Admin Cluster 1 Admin Cluster 2 Admin Topic 1 Resource Owner Topic 2 Resource Owner Dev Read Only - Topic 1 Dev Write - Topic 2 RBAC Authorization Control access to organizations, environments and clusters Admin Roles: ● OrganizationAdmin ● EnvironmentAdmin ● CloudClusterAdmin Control CRUD operations within Kafka resources Developer Roles: ● ResourceOwner ● DeveloperRead ● DeveloperWrite ● DeveloperManage Note: A single user can have multiple roles 13 Operator Roles: ● Operator ● MetricsViewer
- 15. Naming Conventions • RBAC & ACLs can be used together - use RBAC in general as the default to grant access - use ACL in particular cases to deny access • both support prefixed rules • governance - visual attribution - stream governance functionality • choose names unlikely to change over time • think about how naming conventions can be enforced (e.g. CI/CD pipeline) 15
- 16. Demo: Role bindings with Prefixed Rules in Confluent Cloud • Authentication: Confluent Cloud local users • Authorization: RBAC prefixed role bindings • Naming Convention: Team name used as prefix 16
- 17. 17
- 18. How to grant temporary access? (Authentication, Authorization & Naming Conventions) 18
- 20. Platform Limits 20 • given by the infrastructure on which Kafka is deployed • Do you know the limits for your deployment? • Confluent Cloud - hard limits & soft limits - different types of clusters (basic, standard & dedicated) - some limits depend on type of cluster - examples of limits: • RBAC role-bindings • ACLs • throughput
- 21. Authentication Authorization Naming Conventions TEAM COLLABORATION Platform Limits Client Quotas
- 22. Client Quotas • applied on (user, client-ID) or client-ID groups • defined at different levels with order of precedence • quotas: - network bandwidth - request rate • early access feature on Confluent Cloud 22 Quota parameter Cloud Client Quotas Apache Kafka Quotas Apply to Service Accounts User or Client ID Managed by Calling the Confluent Cloud API API Interacting with Kafka Directly Level enforced at Cluster level Broker level
- 23. Authentication Authorization Naming Conventions TEAM COLLABORATION Platform Limits Monitoring Client Quotas
- 24. Metrics • as described in “Kafka: The Definitive Guide” 2nd edition: • proactive vs. reactive measures 24
- 25. Monitoring 25 Apache Kafka® Confluent Platform Confluent Cloud • essentially monitoring a Java application • JMX metrics exposed by Kafka • Confluent Control Center • Metrics Viewer Role • JMX metrics • Confluent Health+ • Metrics API • 3rd party monitoring integration • /export endpoint • Metrics Viewer Role • Confluent Cloud UI • self-managed Confluent Control Center
- 27. Confluent Control Center • Self-managed deployment • Can be connected to Confluent Cloud • Can be used to monitor local Connect cluster. • Allows custom notifications. 27
- 29. Authentication Authorization Naming Conventions TEAM COLLABORATION Platform Limits Chargeback Monitoring Client Quotas
- 30. Chargeback • charging individual cost centers for their share of Kafka cluster usage - flat rate - consumption based • chargeback vs. showback • start with a simple model, which can evolve over time • Confluent Control Center insights • Metrics grouped by Principal ID • content about cost effectiveness by Lyndon Hedderly, Confluent Principal Business Value Consultant 30
- 31. Active Connection Count Example 31 Total client connections (Basic & Standard clusters) Max 1000 Number of TCP connections to the cluster that can be open at one time. Available in the Metrics API as active_connection_count. If you are self-managing Kafka, you can look at the broker kafka.server:type=socket-server- metrics,listener={listener_name},networkProcessor={#},name=connection-count metrics to understand how many connections you are using. This value can vary widely based on several factors, including number of producer clients, number of consumer clients, partition keying strategy, produce patterns per client, and consume patterns per client. To reduce usage on this dimension, you can reduce the total number of clients connecting to the cluster.
- 32. Authentication Authorization Naming Conventions Automation TEAM COLLABORATION Platform Limits Chargeback Monitoring Client Quotas
- 33. Automation • manage infrastructure & resource lifecycle safely & efficiently • easily scale • reuse & abstract • tooling: - command line interface (CLI) & APIs - Confluent for Kubernetes to deploy Confluent Platform on Kubernetes • Quickstart - Terraform provider for Confluent Cloud • sample project • resources & data sources 33
- 34. Terraform Example • Role bindings with prefixed rules • Literal with role binding assignment 34
- 35. Terraform Considerations • starting a new project vs. migrating existing clusters • Decide weather to support all possible options or provide Tshirt-sized templates. • The lifecycle Meta-Argument lifecycle { prevent_destroy = true } 35
- 36. Authentication Authorization Naming Conventions Automation TEAM COLLABORATION Platform Limits Chargeback Monitoring Client Quotas
- 37. Summary • tools & approaches to achieve - scalable & repeatable actions - predictability - self-service capabilities - isolation - manageability • Github Repo with links to documentation & code for examples https://github.com/maaarv/current2022 37
- 38. Q&A 38
- 39. Your Apache Kafka® journey begins here developer.confluent.io 39