Multi-tenant Flink as-a-service with Kafka on Hopsworks
Download as PPTX, PDF1 like385 views
This document provides a summary of a presentation on multi-tenant Flink-as-a-Service on YARN. It discusses how Hopsworks provides first-class support for running Apache Flink jobs and notebooks in a multi-tenant environment on YARN. It simplifies secure use of Apache Kafka in Flink by handling certificate distribution and service discovery. While YARN support for Flink is still a work in progress, Hopsworks addresses many challenges of running Flink in a multi-tenant cloud environment.
![Hops
[Hadoop For Humans]
Join us!
http://github.com/hopshadoop](https://image.slidesharecdn.com/flink-as-a-service-dowling-flink-forward-sep16-161128140002/85/Multi-tenant-Flink-as-a-service-with-Kafka-on-Hopsworks-27-320.jpg)
Multi-tenant Flink as-a-service with Kafka on Hopsworks
- 1. Multi-Tenant Flink-as-a-Service on YARN Jim Dowling Associate Prof @ KTH Senior Researcher @ SICS CEO @ Logical Clocks AB Slides by Jim Dowling, Theofilos Kakantousis Berlin, 13th September 2016 www.hops.io @hopshadoop
- 2. A Polyglot 2
- 3. Polyglot Data Parallel Processing •Stream Processing - Beam/Flink, Spark •ETL/Batch Processing - Spark, MapReduce •SQL-on-hadoop - Hive, Presto, SparkSQL •Distributed ML - SparkML, FlinkML •Deep Learning - Distributed Tensorflow 3
- 4. Flink Standalone good enough for some •Enterprises are polyglot due to economies of scale •Standalone Flink works great for enterprises - Dedicate some servers - Dedicate some SREs 4
- 5. Polyglot Data Parallel Processing In Context 5 Data Processing Spark, MR, Flink, Presto, Tensorflow Storage HDFS, MapR, S3, WAS Resource Management YARN, Mesos, Kubernetes Metadata Hive, Parquet, Authorization, Search
- 6. Flink for the Little Guy •Flink-as-a-Service on Hops Hadoop - Fully UI Driven, Easy to Install •Project-Based Multi-tenancy 6 Hops
- 7. Flink-as-a-Service running on hops.site 7 SICS ICE: A datacenter research and test environment Purpose: Increase knowledge, strengthen universities, companies and researchers
- 9. Hops-YARN Architecture 9 ResourceMgrs NDB Scheduler YARN Client NodeManagers Resource Trackers Heartbeats (70-95%) AM Reqs (5-30%)
- 10. HopsFS Throughput (Spotify Workload) 10 NDB Setup: 8 Nodes using Xeon E5-2620 2.40GHz Processors and 10GbE. NameNodes: Xeon E5-2620 2.40GHz Processors machines and 10GbE.
- 11. HopsFS Metadata Scaleout 11Assuming 256MB Block Size, 100 GB JVM Heap for Apache Hadoop
- 12. Hopsworks 12
- 13. Hopsworks – Project-Based Multi-Tenancy •A project is a collection of - Users with Roles - HDFS DataSets - Kafka Topics - Notebooks, Jobs •Per-Project quotas - Storage in HDFS - CPU in YARN • Uber-style Pricing •Sharing across Projects - Datasets/Topics 13 project dataset 1 dataset N Topic 1 Topic N Kafka HDFS
- 14. Hopsworks – Dynamic Roles 14 Alice@gmail.com NSA__Alice Authenticate Users__Alice Glassfish HopsFS HopsYARN Projects Secure Impersonation Kafka X.509 Certificates
- 15. Look Ma, No Kerberos! •For each project, a user is issued with a X.509 certificate, containing the project-specific userID. •Services are also issued with X.509 certificates. - Both user and service certs are signed with the same CA. - Services extract the userID from RPCs to identify the caller. •Netflix’ BLESS system is a similar model, with short- lived certificates.
- 16. X.509 Certificate Per Project-Specific User 16 Alice@gmail.com Authenticate Add/Del Users Distributed Database Insert/Remove CertsProject Mgr Root CA Services Hadoop Spark Kafka etc Cert Signing Requests
- 17. Flink on YARN •Two modes: detached or blocking •Hopsworks supports detached mode - Client started locally, then exits after the job is submitted to YARN - No accumulator results or exceptions from the ExecutionEnvironment.execute() - Can only kill YARN job, not Flink session. Cleanup issues. •New Architecture proposal for a Flink Dispatcher
- 18. A Flink/Kafka Job on YARN with Hopsworks 18 Alice@gmail.com 1. Launch Flink Job Distributed Database 2. Get certs, service endpoints YARN Private LocalResources Flink/Kafka Streaming App 4. Materialize certs 3. YARN Job + config 6. Get Schema 7. Consume Produce 5. Read Certs Hopsworks KafkaUtil
- 19. Flink Stream Producer in Secure Kafka StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); String topic = parameterTool.get("topic"); 1. Discover: Schema Registry and Kafka Broker Endpoints 2. Create: Kafka Properties file with certs and broker details 3. Create: producer using Kafka Properties 4. Distribute: X.509 certs to all hosts on the cluster 5. Download: the Schema for the Topic from the Schema Registry 6. Do this all securely DataStream<…> messageStream = env.addSource(…); messageStream.addSink(producer); env.execute("Write to Kafka"); 19 Developer Operations
- 20. Flink/Kafka Stream Producer in Hopsworks StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); String topic = parameterTool.get("topic"); FlinkProducer producer = KafkaUtil.getFlinkProducer(topic); DataStream<…> messageStream = env.addSource(…); messageStream.addSink(producer); env.execute("Write to Kafka"); 20https://github.com/hopshadoop/hops-kafka-examples
- 21. Flink/Kafka Stream Consumer in Hopsworks StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); String topic = parameterTool.get("topic"); FlinkConsumer consumer = KafkaUtil.getFlinkConsumer(topic); DataStream<…> messageStream = env.addSource(consumer); RollingSink<String> rollingSink = ... // HDFS path messageStream.addSink(rollingSink); env.execute(“Read from Kafka, write to HDFS"); 21https://github.com/hopshadoop/hops-kafka-examples
- 22. Zeppelin Support for Flink 22
- 23. Karamel/Chef for Automated Installation 23 Google Compute Engine BareMetal
- 24. Demo 24
- 25. Summary •Hopsworks provides first-class support for Flink-as-a-Service - Streaming or Batch Jobs - Zeppelin Notebooks •Hopworks simplifies secure use of Kafka in Flink on YARN •YARN support for Flink still a work-in-progress 25
- 26. Hops Team Active: Jim Dowling, Seif Haridi, Tor Björn Minde, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theofilos Kakantousis, Johan Svedlund Nordström, Konstantin Popov, Antonios Kouzoupis. Ermias Gebremeskel, Daniel Bekele Alumni: Vasileios Giannokostas, Misganu Dessalegn, Rizvi Hasan, Paul Mälzer, Bram Leenders, Juan Roca, K “Sri” Srijeyanthan, Steffen Grohsschmiedt, Alberto Lorente, Andre Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Jude D’Souza, Qi Qi, Gayana Chandrasekara, Nikolaos Stanogias, Daniel Bali, Ioannis Kerkinos, Peter Buechler, Pushparaj Motamari, Hamid Afzali, Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu.
- 27. Hops [Hadoop For Humans] Join us! http://github.com/hopshadoop
Editor's Notes
- #6: Stream Processing Flink ETL workflow (batch) processing Spark SQL-on-hadoop Presto, SparkSQL, Hive Deep Learning Distributed Tensorflow
- #14: Privileges – upload/download data, run analysis jobs Like RBAC solution. All access via HopsWorks.
- #15: 14
- #17: 16
- #18: Netty dependency conflict with our app in blocking mode Impacts: application size, main class run on our multi-tenant application - System.exit(), logs are written locally No accumulator results or exceptions from the ExecutionEnvironment.execute() call Can only kill YARN job, not Flink session – cleanup issues Flink Dispatcher The client directly starts the Job in YARN, rather than bootstrapping a cluster and after that submitting the job to that cluster. The client can hence disconnect immediately after the job was submitted All user code libraries and config files are directly in the Application Classpath, rather than in the dynamic user code class loader Containers are requested as needed and will be released when not used any more The “as needed” allocation of containers allows for different profiles of containers (CPU / memory) to be used for different operators
- #19: 18
- #20: public class HopsKafkaUtil implements Serializable { KAFKA_BROKERADDR_ENV_VAR = "kafka.brokeraddress"; KAFKA_RESTENDPOINT = "kafka.restendpoint"; KAFKA_SESSIONID_ENV_VAR = "kafka.sessionid"; KAFKA_PROJECTID_ENV_VAR = "kafka.projectid"; KAFKA_K_CERTIFICATE_ENV_VAR = "kafka_k_certificate"; KAFKA_T_CERTIFICATE_ENV_VAR = "kafka_t_certificate"; String getHopsConsumer(String topic) {…} String getHopsProducer(String topic) {…} String getHopsFlinkKafkaConsumer(String topic) {…} String getHopsFlinkKafkaProducer(String topic) {…} String getSchema(String topicName, int versionId) {..} Map<String, String> getKafkaProps(String propsStr) {…} }
- #21: HopsKafkaProperties.defaultProps())
- #22: HopsKafkaProperties.defaultProps())
- #23: https://gist.github.com/rawkintrevo/ad206879753733f5a536
- #28: I need some sound-effects to go with that.