SlideShare a Scribd company logo
Flink Security
Enhancements
Eron Wright – eron.wright@emc.com
DELL EMC
@eronwright
2 of 11
New Security Features
1. Kerberos Authentication Support
2. Service-Level Authorization
3. Transport Security (SSL/TLS)
3 of 11
Existing Capability
• Hadoop Delegation Token (DT)
• CLI usesKerberosto authenticateto HDFS
• HDFSprovidesa DT, which CLI passesto the Flinkcluster
• Clusteris ableto accessHDFSfilesonbehalfof theuser
• Limitations
• YARN mode only
• Not usefulto non-Hadoopservices,e.g. Kafka.
• Note: Still supported
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
delegation token
4 of 11
Kerberos Authentication Support
• “Cluster-Level Kerberos Identity”
• Keytab-based
• Sharedby alljobs, notjob-specific
• Enables Kerberos authentication
• DataSourcesandSinks(HDFS,Kafka…)
• StateBackends(ZooKeeper…)
• Protects state data
• ACL onznodes,files
• Supported in standalone and YARN
deployment modes
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
keytab
5 of 11
Service-Level Authorization
• “Restrict access to your Flink cluster”
• Protects all endpoints:
• Akka System(control path)
• Intra-ClusterDataTransfer
• WebUI
• BlobTransfer(JARs…)
• Simple shared secret
• Configuredor generated
• Storedonclient (~/.flink/…)
• Storedincluster
• Supported in standalone and YARN
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTP
Flink
Cluster
keytab secret
6 of 11
Transport-Level Security (SSL/TLS)
• “SSL for all connections”
• May be enabled on a per-endpoint basis
• WebUIis problematic
• Supported in standalone and YARN TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTPS
Flink
Cluster
keytab secret TLS cert(s)
Demo
8 of 11
Configuration
• Configure Kerberos Identity:
– security.enabled: true
– security.keytab: /path/to/keytab
– security.principal: name@realm
• Configure Service-Level Authorization:
– security.cookie: (secret cookie)
• Configure Transport-Level Security:
– security.ssl.enabled: true
– security.ssl.keystore: /path/to/keystore
– security.ssl.keystore-password: (password)
– security.ssl.key-password: (password)
– security.ssl.truststore: /path/to/truststore
– security.ssl.truststore-password: (password)
TM
TM
DATA
AKKA
JM
CLI
WEB
BROWSER
KAFKA HDFSZK
HTTPS
Flink
Cluster
keytab secret TLS cert(s)
Summary
10 of 11
Project Status
• Targeted for: Flink 1.2
• Contributors:
– Vijay Srinivasaraghavan (Dell EMC)
– Suresh Krishnappa (Dell EMC)
• Design Doc: Secure Data Access on Google Docs
• JIRAs:
– FLINK-3929 - Support for Kerberos Authentication with Keytab Credential
– FLINK-3930 - Implement Service-Level Authorization
– FLINK-3931 - Implement Transport Encryption (SSL/TLS)
– FLINK-3932 - Implement State Backend Security
• Code:
– Github: https://github.com/EronWright/flink/tree/feature-flink-security
Eron Wright - Flink Security Enhancements

More Related Content

PDF
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
PPTX
Strata Hadoop Hopsworks
PPTX
Eron Wright - Introducing Flink on Mesos
PDF
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
PDF
Power of the Log: LSM & Append Only Data Structures
PDF
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...
PDF
What's new in Confluent 3.2 and Apache Kafka 0.10.2
PDF
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...
Hopsworks - Self-Service Spark/Flink/Kafka/Hadoop
Strata Hadoop Hopsworks
Eron Wright - Introducing Flink on Mesos
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Power of the Log: LSM & Append Only Data Structures
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...
What's new in Confluent 3.2 and Apache Kafka 0.10.2
Kafka for Microservices – You absolutely need Avro Schemas! | Gerardo Gutierr...

What's hot (20)

PDF
Kafka Summit SF 2017 - Kafka and the Polyglot Programmer
PPTX
Capture the Streams of Database Changes
PDF
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
PPTX
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
PPTX
Apache Kafka 0.8 basic training - Verisign
PDF
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
PPTX
Managing multiple event types in a single topic with Schema Registry | Bill B...
PDF
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
PDF
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
PDF
Apache Pulsar at Yahoo! Japan
PDF
Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture
PPTX
Confluent building a real-time streaming platform using kafka streams and k...
PDF
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
PPTX
Tuning kafka pipelines
PDF
Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019
PDF
Kafka and Spark Streaming
PDF
A Unified Platform for Real-time Storage and Processing
PDF
Query Pulsar Streams using Apache Flink
PDF
Cooperative Data Exploration with iPython Notebook
ODP
Introduction to Apache Kafka- Part 1
Kafka Summit SF 2017 - Kafka and the Polyglot Programmer
Capture the Streams of Database Changes
Tradeoffs in Distributed Systems Design: Is Kafka The Best? (Ben Stopford and...
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
Apache Kafka 0.8 basic training - Verisign
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Managing multiple event types in a single topic with Schema Registry | Bill B...
Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Apache Pulsar at Yahoo! Japan
Gwen Shapira, Confluent | Kafka Summit 2020 Keynote | Kafka’s New Architecture
Confluent building a real-time streaming platform using kafka streams and k...
Shattering The Monolith(s) (Martin Kess, Namely) Kafka Summit SF 2019
Tuning kafka pipelines
Kafka on Kubernetes: Keeping It Simple (Nikki Thean, Etsy) Kafka Summit SF 2019
Kafka and Spark Streaming
A Unified Platform for Real-time Storage and Processing
Query Pulsar Streams using Apache Flink
Cooperative Data Exploration with iPython Notebook
Introduction to Apache Kafka- Part 1
Ad

Viewers also liked (20)

PDF
Márton Balassi Streaming ML with Flink-
PPTX
Stephan Ewen - Scaling to large State
PDF
Julian Hyde - Streaming SQL
PDF
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
PPTX
Gábor Horváth - Code Generation in Serializers and Comparators of Apache Flink
PDF
Trevor Grant - Apache Zeppelin - A friendlier way to Flink
PDF
Alexander Kolb - Flinkspector – Taming the squirrel
PDF
Ana M Martinez - AMIDST Toolbox- Scalable probabilistic machine learning with...
PPTX
Ted Dunning-Faster and Furiouser- Flink Drift
PDF
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in Flink
PPTX
Ted Dunning - Keynote: How Can We Take Flink Forward?
PDF
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
PPTX
Aljoscha Krettek - The Future of Apache Flink
PDF
Zoltán Zvara - Advanced visualization of Flink and Spark jobs

PDF
Jamie Grier - Robust Stream Processing with Apache Flink
PPTX
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
PDF
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
PPTX
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
PPTX
Stephan Ewen - Running Flink Everywhere
PPTX
Robert Metzger - Connecting Apache Flink to the World - Reviewing the streami...
Márton Balassi Streaming ML with Flink-
Stephan Ewen - Scaling to large State
Julian Hyde - Streaming SQL
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
Gábor Horváth - Code Generation in Serializers and Comparators of Apache Flink
Trevor Grant - Apache Zeppelin - A friendlier way to Flink
Alexander Kolb - Flinkspector – Taming the squirrel
Ana M Martinez - AMIDST Toolbox- Scalable probabilistic machine learning with...
Ted Dunning-Faster and Furiouser- Flink Drift
Maxim Fateev - Beyond the Watermark- On-Demand Backfilling in Flink
Ted Dunning - Keynote: How Can We Take Flink Forward?
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Aljoscha Krettek - The Future of Apache Flink
Zoltán Zvara - Advanced visualization of Flink and Spark jobs

Jamie Grier - Robust Stream Processing with Apache Flink
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
Malo Denielou - No shard left behind: Dynamic work rebalancing in Apache Beam
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Stephan Ewen - Running Flink Everywhere
Robert Metzger - Connecting Apache Flink to the World - Reviewing the streami...
Ad

Similar to Eron Wright - Flink Security Enhancements (17)

PPTX
Deploying a secured Flink cluster on Kubernetes
PPTX
Flink Forward Berlin 2018: Edward Alexander Rojas Clavijo - "Deploying a secu...
PDF
Hadoop security
PPTX
Hadoop Security Today and Tomorrow
PPTX
Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!
PPTX
Curb your insecurity with HDP
PPTX
Open Source Security Tools for Big Data
PPTX
Open Source Security Tools for Big Data
PPTX
Hadoop security
PPTX
Hadoop Security Today & Tomorrow with Apache Knox
PPTX
Securing Hadoop in an Enterprise Context
PPTX
Securing Hadoop in an Enterprise Context (v2)
PPTX
Securing the Hadoop Ecosystem
PPTX
Securing Hadoop in an Enterprise Context
PPTX
Hadoop security @ Philly Hadoop Meetup May 2015
PDF
Technical tips for secure Apache Hadoop cluster #ApacheConAsia #ApacheCon
PDF
Curb your insecurity with HDP - Tips for a Secure Cluster
Deploying a secured Flink cluster on Kubernetes
Flink Forward Berlin 2018: Edward Alexander Rojas Clavijo - "Deploying a secu...
Hadoop security
Hadoop Security Today and Tomorrow
Curb Your Insecurity - Tips for a Secure Cluster (with Spark too)!!
Curb your insecurity with HDP
Open Source Security Tools for Big Data
Open Source Security Tools for Big Data
Hadoop security
Hadoop Security Today & Tomorrow with Apache Knox
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise Context (v2)
Securing the Hadoop Ecosystem
Securing Hadoop in an Enterprise Context
Hadoop security @ Philly Hadoop Meetup May 2015
Technical tips for secure Apache Hadoop cluster #ApacheConAsia #ApacheCon
Curb your insecurity with HDP - Tips for a Secure Cluster

More from Flink Forward (20)

PDF
Building a fully managed stream processing platform on Flink at scale for Lin...
PPTX
Evening out the uneven: dealing with skew in Flink
PPTX
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
PDF
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
PDF
Introducing the Apache Flink Kubernetes Operator
PPTX
Autoscaling Flink with Reactive Mode
PDF
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
PPTX
One sink to rule them all: Introducing the new Async Sink
PPTX
Tuning Apache Kafka Connectors for Flink.pptx
PDF
Flink powered stream processing platform at Pinterest
PPTX
Apache Flink in the Cloud-Native Era
PPTX
Where is my bottleneck? Performance troubleshooting in Flink
PPTX
Using the New Apache Flink Kubernetes Operator in a Production Deployment
PPTX
The Current State of Table API in 2022
PDF
Flink SQL on Pulsar made easy
PPTX
Dynamic Rule-based Real-time Market Data Alerts
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
PPTX
Processing Semantically-Ordered Streams in Financial Services
PDF
Tame the small files problem and optimize data layout for streaming ingestion...
PDF
Batch Processing at Scale with Flink & Iceberg
Building a fully managed stream processing platform on Flink at scale for Lin...
Evening out the uneven: dealing with skew in Flink
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing the Apache Flink Kubernetes Operator
Autoscaling Flink with Reactive Mode
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
One sink to rule them all: Introducing the new Async Sink
Tuning Apache Kafka Connectors for Flink.pptx
Flink powered stream processing platform at Pinterest
Apache Flink in the Cloud-Native Era
Where is my bottleneck? Performance troubleshooting in Flink
Using the New Apache Flink Kubernetes Operator in a Production Deployment
The Current State of Table API in 2022
Flink SQL on Pulsar made easy
Dynamic Rule-based Real-time Market Data Alerts
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Processing Semantically-Ordered Streams in Financial Services
Tame the small files problem and optimize data layout for streaming ingestion...
Batch Processing at Scale with Flink & Iceberg

Recently uploaded (20)

PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
PPTX
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
PPTX
Business Acumen Training GuidePresentation.pptx
PPTX
Acceptance and paychological effects of mandatory extra coach I classes.pptx
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
PPT
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
PDF
Clinical guidelines as a resource for EBP(1).pdf
PDF
Recruitment and Placement PPT.pdfbjfibjdfbjfobj
PPTX
Computer network topology notes for revision
PDF
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
PPT
Quality review (1)_presentation of this 21
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
PPTX
Global journeys: estimating international migration
PPT
Miokarditis (Inflamasi pada Otot Jantung)
PPT
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
PDF
Launch Your Data Science Career in Kochi – 2025
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
PPTX
Business Ppt On Nestle.pptx huunnnhhgfvu
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
Data_Analytics_and_PowerBI_Presentation.pptx
CEE 2 REPORT G7.pptxbdbshjdgsgjgsjfiuhsd
Business Acumen Training GuidePresentation.pptx
Acceptance and paychological effects of mandatory extra coach I classes.pptx
STUDY DESIGN details- Lt Col Maksud (21).pptx
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
Chapter 2 METAL FORMINGhhhhhhhjjjjmmmmmmmmm
Clinical guidelines as a resource for EBP(1).pdf
Recruitment and Placement PPT.pdfbjfibjdfbjfobj
Computer network topology notes for revision
TRAFFIC-MANAGEMENT-AND-ACCIDENT-INVESTIGATION-WITH-DRIVING-PDF-FILE.pdf
Quality review (1)_presentation of this 21
oil_refinery_comprehensive_20250804084928 (1).pptx
Global journeys: estimating international migration
Miokarditis (Inflamasi pada Otot Jantung)
Chapter 3 METAL JOINING.pptnnnnnnnnnnnnn
Launch Your Data Science Career in Kochi – 2025
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
Business Ppt On Nestle.pptx huunnnhhgfvu
iec ppt-1 pptx icmr ppt on rehabilitation.pptx

Eron Wright - Flink Security Enhancements

  • 1. Flink Security Enhancements Eron Wright – eron.wright@emc.com DELL EMC @eronwright
  • 2. 2 of 11 New Security Features 1. Kerberos Authentication Support 2. Service-Level Authorization 3. Transport Security (SSL/TLS)
  • 3. 3 of 11 Existing Capability • Hadoop Delegation Token (DT) • CLI usesKerberosto authenticateto HDFS • HDFSprovidesa DT, which CLI passesto the Flinkcluster • Clusteris ableto accessHDFSfilesonbehalfof theuser • Limitations • YARN mode only • Not usefulto non-Hadoopservices,e.g. Kafka. • Note: Still supported TM TM DATA AKKA JM CLI WEB BROWSER KAFKA HDFSZK HTTP Flink Cluster delegation token
  • 4. 4 of 11 Kerberos Authentication Support • “Cluster-Level Kerberos Identity” • Keytab-based • Sharedby alljobs, notjob-specific • Enables Kerberos authentication • DataSourcesandSinks(HDFS,Kafka…) • StateBackends(ZooKeeper…) • Protects state data • ACL onznodes,files • Supported in standalone and YARN deployment modes TM TM DATA AKKA JM CLI WEB BROWSER KAFKA HDFSZK HTTP Flink Cluster keytab
  • 5. 5 of 11 Service-Level Authorization • “Restrict access to your Flink cluster” • Protects all endpoints: • Akka System(control path) • Intra-ClusterDataTransfer • WebUI • BlobTransfer(JARs…) • Simple shared secret • Configuredor generated • Storedonclient (~/.flink/…) • Storedincluster • Supported in standalone and YARN TM TM DATA AKKA JM CLI WEB BROWSER KAFKA HDFSZK HTTP Flink Cluster keytab secret
  • 6. 6 of 11 Transport-Level Security (SSL/TLS) • “SSL for all connections” • May be enabled on a per-endpoint basis • WebUIis problematic • Supported in standalone and YARN TM TM DATA AKKA JM CLI WEB BROWSER KAFKA HDFSZK HTTPS Flink Cluster keytab secret TLS cert(s)
  • 8. 8 of 11 Configuration • Configure Kerberos Identity: – security.enabled: true – security.keytab: /path/to/keytab – security.principal: name@realm • Configure Service-Level Authorization: – security.cookie: (secret cookie) • Configure Transport-Level Security: – security.ssl.enabled: true – security.ssl.keystore: /path/to/keystore – security.ssl.keystore-password: (password) – security.ssl.key-password: (password) – security.ssl.truststore: /path/to/truststore – security.ssl.truststore-password: (password) TM TM DATA AKKA JM CLI WEB BROWSER KAFKA HDFSZK HTTPS Flink Cluster keytab secret TLS cert(s)
  • 10. 10 of 11 Project Status • Targeted for: Flink 1.2 • Contributors: – Vijay Srinivasaraghavan (Dell EMC) – Suresh Krishnappa (Dell EMC) • Design Doc: Secure Data Access on Google Docs • JIRAs: – FLINK-3929 - Support for Kerberos Authentication with Keytab Credential – FLINK-3930 - Implement Service-Level Authorization – FLINK-3931 - Implement Transport Encryption (SSL/TLS) – FLINK-3932 - Implement State Backend Security • Code: – Github: https://github.com/EronWright/flink/tree/feature-flink-security