Hadoop 3 (2017 hadoop taiwan workshop)
Download as PPTX, PDF3 likes568 views
The document discusses new features in Apache Hadoop 3, including HDFS erasure coding which reduces storage overhead, YARN federation which improves scalability, and the Application Timeline Server which provides improved visibility into application performance. It also covers HDFS multi standby NameNodes which enhances high availability, and the future directions of Hadoop including object storage with Ozone and running HDFS on cloud infrastructure.
Hadoop 3 (2017 hadoop taiwan workshop)
- 1. © Cloudera, Inc. All rights reserved. 1 Hadoop 3 is coming â whatâs new and whatâs next?
- 2. © Cloudera, Inc. All rights reserved. 2 About Wei-Chiu Apache Hadoop committer Software Engineer, Cloudera
- 3. © Cloudera, Inc. All rights reserved. 3 Agenda The Problem What is Hadoop Major Hadoop 3 Features Whatâs Next?
- 4. © Cloudera, Inc. All rights reserved. 4 - Anne Wojcicki âData helps solve problemsâ
- 5. © Cloudera, Inc. All rights reserved. 5 Big Data - 3Vs Volume Velocity Variety
- 6. © Cloudera, Inc. All rights reserved. 6 Apache Hadoop The de facto Big Data Analytics platform A distributed framework to support large scale computation on commodity hardware ⹠Petabyte+ storage, 1,000+ compute nodes ⹠Inspired by Google ⹠Originally developed by Yahoo!, donated to Apache Software Foundation. ⹠Open source :) ⹠183 committers, thousands contributors
- 7. © Cloudera, Inc. All rights reserved. 7 Apache Hadoop
- 8. © Cloudera, Inc. All rights reserved. 8 Cloudera Commercializes Hadoop* technology Open source, open culture CDH - Clouderaâs Distribution for Hadoop âą Platform. Open source Cloudera Manager (CM), Cloudera Navigator, Key Trustee âą Cluster management, monitoring. Proprietary *Hadoop and its associated projects
- 9. © Cloudera, Inc. All rights reserved. 9 Hadoop Ecosystem
- 10. © Cloudera, Inc. All rights reserved. 10 Storage: reduce storage cost Compute: much larger cluster Well, data itself is a problem ... High availability High performance Seamless experience More cloud usage Ease of development Clusters becoming larger More enterprise adoption New applications
- 11. © Cloudera, Inc. All rights reserved. 11 HDFS Erasure Coding
- 12. © Cloudera, Inc. All rights reserved. 12 Hadoop Distributed File System 3x replication write read A fault tolerant, highly scalable storage system POSIX semantics Security â user authentication, authorization, at-rest encryption, transport encryption NameNode DataNode
- 13. © Cloudera, Inc. All rights reserved. 13 Advantage ⹠Failure tolerant But ⹠3x storage cost ⹠3x datacenter space ⹠3x power consumption Hadoop Distributed File System How to reduce storage overhead?
- 14. © Cloudera, Inc. All rights reserved. 14 âąParity bit âąXOR âąIf X is lost, X can be reconstructed using Y and X ^ Y âą50% overhead ((3-2)/2) âąCan tolerate one failure âąReedâSolomon âąRS(k,m) tolerates m failures in k data cells. âąXOR = RS(2,1) Erasure Coding 101
- 15. © Cloudera, Inc. All rights reserved. 15 Reed-Solomon ⹠Compute parity bits for redundancy ⹠Blocks can be reconstructed after failures ⹠Configurable durability v.s. storage overhead RS(6,3) ⹠= 50% storage overhead ⹠(9-6)/6 Erasure Coding 101 K blocks encode M parity blocks
- 16. © Cloudera, Inc. All rights reserved. 16 HDFS-EC: RS(6,3) write read block strip 1 strip 2 strip 3 strip 4 strip 5 strip 6 parity 1 parity 2 parity 3
- 17. © Cloudera, Inc. All rights reserved. 17 HDFS-EC: Failure Handling
- 18. © Cloudera, Inc. All rights reserved. 18
- 19. © Cloudera, Inc. All rights reserved. 19 YARN Federation
- 20. © Cloudera, Inc. All rights reserved. 20 YARN A resource management framework for Hadoop clusters â Highly scalable, 4000 - 8000 nodes in production â Hive, Oozie, Spark, ⊠â HBase
- 21. © Cloudera, Inc. All rights reserved. 21 YARN Resource Manager Application Master Client Node Manager
- 22. © Cloudera, Inc. All rights reserved. 22 YARN Federation Developed by Microsoft Extreme scale â 100,000 compute nodes â Resource Manager becomes the bottleneck
- 23. © Cloudera, Inc. All rights reserved. 23 YARN Federation Application Master Application Master RM Proxy RM 1 RM 2
- 24. © Cloudera, Inc. All rights reserved. 24 YARN Timeline Service v2
- 25. © Cloudera, Inc. All rights reserved. 25 Job History Server âKeeps track of job progress â Collect or retrieve information of MapReduce jobs âExtensibility âMR only âUsability âNo YARN level events âMetrics can only be retrieved after job terminates
- 26. © Cloudera, Inc. All rights reserved. 26 Application Timeline server v2 â Development led by Twitter â Usability â Flow: logical group of applications â Scalability â HBase â Use cases â Analyze application performance. â Cluster capacity planning.
- 27. © Cloudera, Inc. All rights reserved. 27 Use HBase for storage Use cases: â Analyze application performance. â Cluster capacity planning. ASTv2 Architecture
- 28. © Cloudera, Inc. All rights reserved. 28 HDFS Multi Standby NameNodes
- 29. © Cloudera, Inc. All rights reserved. 29 NameNode High Availability Active NameNode Standby NameNode Journal Node Journal Node Journal Node Quorum Client Upload fsimage
- 30. © Cloudera, Inc. All rights reserved. 30 Contributed by Salesforce. Multiple Standby NameNode Active NameNode Standby NameNode Journal Node Journal Node Journal Node Quorum Client Standby NameNode Upload fsimage Upload fsimage
- 31. © Cloudera, Inc. All rights reserved. 34 Classpath Isolation
- 32. © Cloudera, Inc. All rights reserved. 35 Dependency Hell
- 33. © Cloudera, Inc. All rights reserved. 36 Dependency Hell Hadoop was not initially designed as foundation of many applications. â More applications depending on Hadoop â harder for Hadoop to upgrade dependency libraries. â Potential risk to break existing applications â Increase exposure to security vulnerabilities Classpath Isolation â Separate client-side classpath from server-side
- 34. © Cloudera, Inc. All rights reserved. 37 Cloud
- 35. © Cloudera, Inc. All rights reserved. 38 â Cloud connectors â Microsoft Azure Data Lake filesystem â Aliyun Object Storage Service Other features
- 36. © Cloudera, Inc. All rights reserved. 39 Misc.
- 37. © Cloudera, Inc. All rights reserved. 40 â Shell script rewrite â Requires Java 8 â Server ports â Remove legacy features â S3 file system â S3A (recommended) or S3N â Hftp â webhdfs/httpfs â Bookkeeper Journal Manager â Quorum Journal Manager Other features and incompatibility
- 38. © Cloudera, Inc. All rights reserved. 41 Whatâs next?
- 39. © Cloudera, Inc. All rights reserved. 42 Developers ⹠Use it early, test it early and file bug reports. Administrators ⹠Test upgradability Users ⹠Expect better user experience. Now what? Alpha 1 2016/09 2016/12 Alpha 2 Beta 1 2017/0? GA ? CDH6 Hadoop 3 Timeline 2017/01 Alpha 3 2017/0?
- 40. © Cloudera, Inc. All rights reserved. 43 â We donât know yet. â Ozone (HDFS-7240) â Object store for HDFS â HDFS over cloud (HDFS-9806) â Emerging applications and use cases â Docker â Deep learning â Hardware Trend â Cloud storage â Faster ethernet (40GBps), high density (> 100TB) storage node â Memory technology â Locality will not be a deciding factor. Future? Hadoop 4?
- 41. © Cloudera, Inc. All rights reserved. 44 Ozone (HDFS-7240) Status quo â NameNode is becoming a bottleneck â A general file system may not suit the specific need of an application Solution â Split HDFS namespace into blob stores
- 42. © Cloudera, Inc. All rights reserved. 45 HDFS over Cloud (HDFS-9806) Use case â Use HDFS for temporary data â Use cloud for permanent storage The problem â Data management â Consistency Solution â HDFS as metastore and cache â Cloud as backend data store
- 43. © Cloudera, Inc. All rights reserved. 46 Ask Bigger Questions
- 44. © Cloudera, Inc. All rights reserved. 47 ⹠Introduction to HDFS Erasure Coding in Apache Hadoop ⹠Enable YARN RM scale out via federation using multiple RM's ⹠Application Timeline Server - Past, Present and Future ⹠HDFS-6440 Support more than 2 NameNodes ⹠How-to: Use the New HDFS Intra-DataNode Disk Balancer in Apache Hadoop References
Editor's Notes
- #5: We live in information era. We live in a big data era.
- #6: More important than 3 Vs, is analyze the data, and get extra value out of it. Leverage big data to answer new questions.
- #7: Compare to MPI: data locality, embarrasingly parallel, fault tolerant, commodity hardware Modeled after Googleâs Google File System and MapReduce.
- #8: Hadoop cluster at Yahoo
- #9: Health care, telecom, retail, tech, media, government,
- #10: This is Clouderaâs view of the Hadoop ecosystem. Of course, other Hadoop distributions or platform distributions may have other views of the system. (Confluence, Databricks, Datastax
- #11: I am not talking about big data application, but big data management problem.
- #13: Use commodity hardware means failure is a norm. Must handle it. HDFS is a distributed file system -- high throughput Traditionally replicated block architecture
- #14: As cluster grows, there are more cold data. These cold data doesnât need high through. They just need redundancy.
- #16: http://blog.cloudera.com/blog/2015/09/introduction-to-hdfs-erasure-coding-in-apache-hadoop/
- #17: Traditionally replicated block architecture
- #19: The encoding/deciding of EC requires hardware acceleration ISA-L (4x speedup). Intel folks has contributed significantly in this project. Compare Facebookâs HDFS-RAID coder, pure Java coder, ISA-L coder
- #21: Think of operating system kernel scheduler that runs multiple processes on multicore. YARN-2915 https://issues.apache.org/jira/browse/YARN-2915
- #22: Think of operating system kernel scheduler that runs multiple processes on multicore. YARN-2915 https://issues.apache.org/jira/browse/YARN-2915
- #25: http://www.slideshare.net/NaganarasimhaGarla/application-timeline-server-past-present-and-future-53027238
- #26: Each application has its own JHS. Information is not shared between each JHS. It is hard to trace an oozie workflow, say it starts a spark job, runs hdfs command, start a yarn app. Very little visbility of global picture. metrics is stored in HDFS. Can only be retrieved when job terminates.
- #29: Single point of failure HDF-6440, contributed by Salesforce
- #31: Donât go more than 3~5 nematodes in practice.
- #39: s3n will not be maintained other than security fix
- #41: s3n will not be maintained other than security fix
- #44: Spend 10 minutes Hadoop was based on the assumption to move compute to data 3D XPoint memory technology Virtualized compute, storage and networks
- #45: Hortonworks
- #46: Microsoft
- #47: Ask bigger questions