Kafka Security 101 and Real-World Tips
7 likes6,843 views
This document provides a comprehensive overview of implementing security in Apache Kafka, highlighting the importance of encryption, authentication, and authorization. It outlines various security mechanisms like SSL, SASL, and how to manage access control lists (ACLs), along with practical tips for real-world applications. The author emphasizes the need for thorough setup and consistent security practices to ensure Kafka's integrity within an organization's infrastructure.
Kafka Security 101 and Real-World Tips
- 1. Kafka Security 101 & Real World Tips Stephane Maarek - DataCumulus
- 2. My Kafka Security Journey Stephane, implement Kafka Security!
- 3. Who am I? • I’m Stephane! • Consultant & Solution Architect at DataCumulus • Apache Kafka SeriesVideo Courses on Udemy • Full productions deployments (with security) • You can find me on • GitHub: https://github.com/simplesteph • LinkedIn: https://www.linkedin.com/in/stephanemaarek • Medium: https://medium.com/@stephane.maarek • Twitter: https://twitter.com/stephanemaarek • Udemy: https://udemy.com/stephane-maarek
- 4. Who has not secured Kafka?
- 5. Kafka without Security is RISKY 5 disastrous scenarios 1. Read all your data 2. Write to any topic and break your consumers 3. Intercept and read plaintext network packets 4. Delete all your Kafka data in one command without SSH 5. Kafka Connect? Database Credentials are in a Kafka Topic, plaintext
- 6. You need Kafka Security If you intend to make Kafka a cornerstone of your infrastructure
- 7. What’s Kafka Security? Disclaimer: the source of truth is always the documentation
- 8. Kafka Security in three words Encryption Authentication Authorization
- 9. Encryption in Kafka • SSL encryption = secure communications • Similar to HTTPS Super secret message Kafka Client (producer / consumer ) Kafka Brokers Port 9093 - SSL aGVsbG8gd29 ybGQgZWh… Encrypted data Kafka Client (producer / consumer ) Kafka Brokers Port 9092 - PLAINTEXT
- 10. SSL, Concretely? • Create a Certificate Authority (CA) • Generate certificates for your brokers, sign them • Make sure your broker and clients trust the CA Root. ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks ssl.keystore.password=serversecret ssl.key.password=serversecret ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks ssl.truststore.password=serversecret
- 11. SSL in the Real World • SSL lowers the performance of your brokers • You lose the zero-copy optimization • Kafka heap usage increases • CPU usage increases • SSL only allows to encrypt data in flight • Data at rest sits un-encrypted on Kafka Disk
- 12. What about end-to-end encryption? • Closed source: Apple • Open source: • https://github.com/Quicksign/kafka-encryption • https://github.com/nucypher/kafka-oss • POC in progress at DataCumulus Producer Kafka PLAINTEXT Consumer Encrypted data Encrypted data encrypt data decrypt data
- 14. Authentication in Kafka • Clients need to have and prove their identity • ~= Login (username / password or token) Kafka Client Kafka Broker Authentication data Verify authentication Client is authenticated
- 15. 99 Forms Of Authentication But Easy Ain’t One • SSL Authentication: two way client authentication • SASL (Simple Authentication and Security Layer): • SASL/GSSAPI (Kerberos) – v0.9.0.0+ - Enterprises (Microsoft AD) • SASL/PLAIN – v0.10.0.0+ - Passwords hardcoded in broker • SASL/SCRAM-SHA-256/512 – v0.10.2.0+ - Passwords in Zookeeper (secure it) • SASL/OAUTHBEARER – v2.0+ - Leverage OAuth 2 • Write your own (contribute back!) • Extend SASL/PLAIN and SASL/SCRAM with KIP-86 (change credentials store) • Real world advice: choose the authentication mechanism you already have in your enterprise
- 16. Take-aways from the battlefield • SSL authentication makes it really hard to revoke authentication • SASL (Simple Authentication and Security Layer) is not simple (YMMV) • Kerberos is by far the hardest to setup right. Errors are cryptic • This is the most challenging part of the Kafka security journey
- 17. Authentication in Kerberos, concretely? 1. Create Kerberos or use Active Directory 2. Ensure Kafka servers have correct CNAME & hostname 3. Generate credentials for the brokers 4. Generate KeyTabs for the brokers from the credentials 5. Create a JAAS file: KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="/tmp/kafka.service.keytab" principal="kafka/<<KAFKA-SERVER-INTERNAL-DNS>>@KAFKA.SECURE"; };
- 18. Authentication in Kerberos, concretely? Continued… • Start Kafka and use java options to reference JAAS file • Add properties to Kafka: • Start Kafka • Pray ! advertised.listeners=SASL_SSL://<<KAFKA-SERVER-DNS>>:9094 sasl.enabled.mechanisms=GSSAPI sasl.kerberos.service.name=kafka
- 19. Real-World Tips: Authentication • Turn on DEBUG log during setup! • Go slow • Ensure CNAME and PTR records are correct • Scrape and repeat can sometimes solve issues • Automate
- 21. Authorisation in Kafka • Kafka knows our client’s identity • + Authorization rules: • ”User alice can read topic finance” • ”User bob cannot write topic trucks” • = Security • ACL (Access Control Lists) have to be maintained by administrators
- 22. ACLs, where are they? • Default:ACLs are stored in Zookeeper • Must secure Zookeeper (network rules or authentication) • OR write your own authorizer (AD, LDAP, a database, Kafka…)
- 23. Managing ACLs Producers • Adding Permissions: • Shortcuts for producer: ~/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181 --add --allow-principal User:myproducer --operation Write --topic mytopic ~/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181 --add --allow-principal User:myproducer --producer --topic mytopic
- 24. Managing ACLs Consumers • Adding Permissions: • Shortcut for consumers: ~/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181 --add --allow-principal User:myconsumer --operation Read --topic mytopic ~/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181 --add --allow-principal User:writer --consumer --topic mytopic ~/kafka/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=<<ZOOKEEPER-DNS>>:2181 --add --allow-principal User:myconsumer --operation Write --group mygroup
- 25. Managing ACLs at Scale? • Look into Kafka Security Manager (https://github.com/simplesteph/kafka-security-manager )
- 26. Real World Tips on ACLs • Authorisation denials will be logged as INFO in the Kafka log. • Define your broker as super users • Careful with: allow.everyone.if.no.acl.found=true • ACLs can be applied to: • Topics: Create, Read, Describe,Write, etc… • Groups: Read,Write, Describe • Cluster: DescribeConfigs,AlterConfigs, Create • Wildcards are supported in Kafka 2.0! (useful for Kafka Streams)
- 27. Cluster Security Broker Broker Broker ZookeeperSASL_SSL SASL Clients SASL_SSL Kafka Cluster Zookeeper Cluster
- 28. Kafka Server is Secured ! Done? Encryption Authentication Authorization
- 29. Security Journey Continued… Stephane, secure Kafka Clients!
- 31. Kafka Client Security is the real challenge • Technical Challenge: • Java Clients: easy • Non Java Clients: please use a client that wraps librdkafka • People Challenge: • Kafka Administrator: I’m a security guru! But I don’t want to secure all the apps • Kafka Developer: wt* is security?
- 32. Client Security in Java security.protocol=SASL_SSL sasl.kerberos.service.name=kafka ssl.truststore.location=/home/kafka/ssl/kafka.client.truststore.jks ssl.truststore.password=clientpass sasl.jaas.config='com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true useTicketCache=false keyTab="/etc/security/keytabs/client.service.keytab" principal="clientusername";' • It’s not fun • It’s not easy • It’s error prone
- 33. Kafka Clients are not tailored to your security needs • Observation 1: Default Kafka clients have every options for security • Observation 2: Your enterprise will only have one security setup • Observation 3: Every client security configuration will look the same • Take-away: don’t use the default Kafka clients
- 34. Real World Advice #1 Distribute your own wrapped Kafka Clients • Developers love nice APIs: • Standardized applications • No copy and paste errors • Centralized debugging • Reduced learning curve for devs new MyCorpKafkaProducer(bootstrapServers, keySerializer, valueSerializer) .withSSL(sslEnabled, pathToTrustStore) .withAuth(authEnabled, pathToKeyTab, usernameOrPrincipal) .withSchemaRegistry(url) .withExtraProperties(properties) .build()
- 35. Real World Advice #2 Create a Kafka Client Base Docker Image Security at scale goes hands in hands with consistency. • Embed modified Java Truststore • Standard Retrieval of SSL certificates • Standard Retrieval of Credentials fromVault / Secure Store • Kafka environment switches, Security switches • Bootstrap Server Discovery & Schema Registry Discovery • Extend to Kafka Connect & Schema Registry
- 36. Real World Advice #3 Make a checklist before going to prod • What’s the application username? • Are all ACLs listed and created? • Is the application using the MyCorp Kafka clients? • Is the application running in the standardized Docker Container? • Are quotas defined for this application? • Is the application monitored? • …Check? Release!
- 37. Next steps Where to take your learning from here!
- 38. Okay, I want to implement security! What’s next? • Read the docs: • Kafka Documentation: https://kafka.apache.org/documentation/#security • Confluent Documentation: https://docs.confluent.io/current/security.html • Read some blogs: • https://medium.com/@stephane.maarek/introduction-to-apache-kafka-security- c8951d410adf • https://www.confluent.io/blog/apache-kafka-security-authorization-authentication- encryption/ • Video Course: • ConfluentYoutube: https://www.youtube.com/watch?v=MsQo-yoVleU&t=21s • Udemy: https://www.udemy.com/apache-kafka-security (coupon KAFKASUMMIT18)