Shug meetup Hops Hadoop
Download as PPTX, PDF1 like146 views
This document discusses a multi-tenant Hadoop-as-a-Service platform called HopsWorks that is available for free use at the SICS ICE research facility in Luleå, Sweden. Key points: - SICS ICE is the world's first open data center dedicated to big data research, with resources like Hadoop/Spark/Flink available as a service. - HopsWorks provides true multi-tenancy for Hadoop through project-specific user IDs and group IDs to isolate data and enforce access controls. - Metadata is kept consistent through the use of a distributed database, with foreign keys ensuring integrity when projects or data sets are modified or deleted. - A
![Access Control in Hadoop: Apache Sentry
16
How do you ensure the consistency of the policies and the data?
[Mujumdar’15]](https://image.slidesharecdn.com/shug-meetup-hadoop-as-a-service-april-2016-161128140736/85/Shug-meetup-Hops-Hadoop-16-320.jpg)
Shug meetup Hops Hadoop
- 1. Multi-Tenant Hadoop-as-a-Service (for free!) Jim Dowling Associate Prof @ KTH Senior Researcher @ SICS CEO @ Hops AB SHUG Meetup, Stockholm, April 21st 2016 www.hops.io @hopshadoop (Some Slides by Prof. Tor Björn Minde, CEO SICS North Swedish ICT AB)
- 3. Talk Overview •World’s First Open Data Centre for Big Data in Luleå •Metadata in Hadoop •True Multi-Tenancy for Hadoop •DEMO: Spark/Flink/Hadoop-as-a-Service 3
- 4. Vision SICS ICE research facility 4 A 2 MW datacenter research and test environment Purpose: Increase knowledge, strengthen universities, companies and researchers R&D institute, 5 lab modules, 3-4000 servers, 2-3000 square meters
- 5. What SICS ICE will offer 1. Compute capacity and tools for big data and cloud • Hadoop/Spark/Flink-as-a-Service 2. Demonstration space for new products & solutions 3. Datacenter infrastructure for experiments and facility data • Flexible lab modules and re-configuration • Measurement equipment for energy, cooling, capacity 4. Competence for verticals and datacenter infrastructure 5
- 6. Status of SICS-ICE research facility (ICE = Infrastructure and Cloud research Environment) Phase 1 (1 room built) • Establish test projects in a “room-in- room” commercial co-location facility • Start of operation February 2016 • Officially Launched in April 2016 Phase 2 (Design phase) • Design of a flexible and general research facility summer-fall 2016 • Contracts with Akademiska Hus & E.ON • Plan is to start build phase Spring 2017 • Plan is to start installation fall 2017 • Plan is to start operation early 2018 6 SICS-ICE Phase1
- 7. Phase 1 room-in-room module 1 7
- 8. A Data Center Optimized for Hadoop 8 Dell servers from Hi5 in module 1 • 3600 cores • 40 TB RAM • Up to 7.5 petabyte storage • 10/40 Gb/s network • Separate management network
- 9. Hadoop-as-a-Service on SICS ICE 9
- 10. But First…. MetaData in Hadoop 10
- 11. Metadata Totem Poles in Hadoop 11Eventual Consistency
- 12. With Many Hadoop Clusters 12 Cluster 1 Cluster N MetaData Service MetaData Service MetaData Service (Aggregator) Eventually consistent MetaData aggregated using more eventually consistent protocols.
- 13. MetaData in Hops Hadoop HDFS YARN NDB Projects DataSets Users ProvenanceSearch HistoryCustomMetaData 13
- 14. Case Study: Access Control as a MetaData Service 14
- 15. Access Control in Relational Databases # Multi-tenancy for alice and bob on db1 and db2 grant all privileges on db1.* to ‘alice'@‘%‘; grant all privileges on db2.* to ‘bob'@‘%‘; #More fine-grained privileges grant SELECT privileges on db2.sensitiveTable to ‘alice'@‘192.168.1.2‘; 15 Databases ensure the consistency of security and policies using foreign keys. “drop table db2.sensitiveTable” => delete associated privileges
- 16. Access Control in Hadoop: Apache Sentry 16 How do you ensure the consistency of the policies and the data? [Mujumdar’15]
- 17. Policy Editor for Sentry 17 Administrators administer privileges for users
- 18. Problem: Sensitive Data needs its own Cluster 18 NSA DataSet User DataSet Alice can copy/cross-link between data sets Alice has only one Kerberos Identity. Neither attribute-based access control nor dynamic roles supported in Hadoop. Alice
- 19. Solution: Project-Specific UserIDs 19 Project NSA Project Users Member of NSA__Alice Users__Alice Member of HDFS enforces access control How can we share DataSets between Projects?
- 20. Sharing DataSets with HopsWorks 20 Project NSA Project Users Member of DataSetowns Add members of Project NSA to the DataSet group NSA__Alice Users__Alice Member of
- 21. Web Application Enforces Dynamic Roles 21 Alice@gmail.com NSA__Alice Authenticate Users__Alice HopsWorks HopsFS HopsYARN Projects Secure Impersonation
- 22. User •Authentication Provider - JDBC Realm - 2-Factor Authentication - LDAP 22
- 23. Project •Users - Roles: Owner, Data Scientist •DataSets - Home project - Can be shared 23
- 24. Project Roles •Data Owner Privileges - Import/Export data - Manage Membership - Share DataSets •Data Scientist Privileges - Write code - Run code - Request access to DataSets 24 We delegate administration of privileges to users
- 25. Per Project CPU and Storage Quotas •300 GB per Project •1000 CPU mins •Uber-Style Pricing - Elastic Demand Curve 25
- 26. Sharing DataSets between Projects 27 The same as Sharing Folders in Dropbox
- 27. Delegate Access Control to HDFS •HDFS enforces access control - UserID per Project - GroupID per Project and DataSet •Metadata Integrity using Foreign Keys - Removing a project removes all users, groups, extended metadata, and (optionally) DataSets. 28
- 28. Free Text Search with Consistent Metadata 29 Free-Text Search Distributed Database ElasticSearch The Distributed Database is the Single Source of Truth. Foreign keys ensure the integrity of Metadata. MetaData Designer MetaData Entry
- 29. The NoteBook Proxy Wars 30
- 30. Demo 31
- 31. Short-Term RoadMap •Multi-tenant Kafka - Per-project Topics •Oozie Workflow Editor •Genomics Support with Adam/Spark •Tiered Storage: Hot Data, Normal, Archived •Improved Data Ingress - Sharing Public DataSets Globally using P2P technology 32
- 32. The Team Active: Jim Dowling, Seif Haridi, Tor Björn Minde, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Kamal Hakimzadeh, Ermias Gebremeskel, Theofilos Kakantousis, Johan Svedlund Nordström, Someya Sayeh, Vasileios Giannokostas, Antonios Kouzoupis, Misganu Dessalegn, Rizvi Hasan, Ahmad Al-Shishtawy, Ali Gholami, Paul Mälzer. Alumni: K. “Sri” Srijeyanthan, Steffen Grohsschmiedt, Alberto Lorente, Andre Moré, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Jude D’Souza, Qi Qi, Gayana Chandrasekara, Nikolaos Stanogias, Daniel Bali, Ioannis Kerkinos, Peter Buechler, Pushparaj Motamari, Hamid Afzali, Wasif Malik, Lalith Suresh, Mariano Valles, Ying Lieu. Hops
- 33. Conclusions •HopsWork is providing a world’s first: Hadoop-as-a- Service to researchers and industry. •Workshop on 12th May, 17.30 – 20.00 in SICS, 6th Floor of the Electrum Building, Kista. Register at www.hops.io/?q=news •Join the team – talk to me! 34 www.hops.io www.hops.site
Editor's Notes
- #2: Associate Prof @ KTH Senior Researcher @ SICS This is an old Brewery – which is a synonym for “Hopworks”, the new product I will talk about today. I am going to talk to you about a product we have been developing for the last few years. It’s “Hadoop for Humans”. We support the main Data Parallel Processing Platforms for Hadoop. And we see Apache Flink as one of those platforms. We expect there to be an ecosystem of platforms, rather than a one-size-fits-all platform. Anyhow, I have been coordinating a large pan-European project called ‘BiobankCloud’ for the last 3 years and one of the problems we have been trying to solve is: how can different medical studies with sensitive data safely reside in the same Hadoop platform. Studies need to be fully isolated from one another That is, we want true multi-tenancy in Hadoop. Now, I am going to try and convince you that HopsWorks provides scaleable multi-tenancy, and other distros don’t. I will try with both rational argument, and more classical approaches. Let’s start with a classical approach. That is – marketing. Marketing people say you won’t believe me. Even if I have a “Dr” before my name. Not good enough. They say, one way around this problem is to get celebrity endorsements. Now, we have a picture of Steve Irwin, the crocodile hunter, on our website, but we couldn’t get him – for obvious reasons… Now here’s our “Logo”
- #3: Spotify is not the only group in Sweden building a next generation platform for Big Data.
- #6: Support network of analytics and cloud researchers Researchers & engineers for datacenter infrastructure support
- #13: 2 main reasons to have separate Hadoop Clusters: Scale and Security. Scale: we support 5x bigger clusters. Separate Clusters for Sensitive DataSets. Sharing data is copying data. Control going down is problematic, as you are making decisions often based on stale data.
- #14: weirwood tree
- #16: What happens to the privileges if I remove “db2.sensitiveTable”? foreign keys maintain integrity of policies and data
- #17: What’s wrong with this picture? No Spark or Flink!
- #18: General Data Protection Regulation act says “Data Owners” should administer privileges – not sysadmins
- #19: Dynamic Roles is what I am talking about for the white-hat and black-hats out there.
- #20: Dynamic Roles is what I am talking about for the white-hat and black-hats out there.
- #21: Dynamic Roles is what I am talking about for the white-hat and black-hats out there.
- #22: Dynamic Roles is what I am talking about for the white-hat and black-hats out there. We use secure impersonation in Hadoop to access HDFS and launch YARN jobs
- #24: Privileges – upload/download data, run analysis jobs Like RBAC solution. All access via HopsWorks.
- #25: Fixed set of privileges for the roles No need for administrator to manage roles -> Users mapping
- #29: Convention for directories