Hadoop, Hive, Spark and Object Stores

Editor's Notes

• #3: Now people may be saying "hang on, these aren't spark developers". Well, I do have some integration patches for spark, but a lot of the integration problems are actually lower down: -filesystem connectors -ORC performance -Hive metastore Rajesh has been doing lots of scale runs and profiling, initially for Hive/Tez, now looking at Spark, including some of the Parquet problems. Chris has done work on HDFS, Azure WASB and most recently S3A Me? Co-author of the Swift connector. author of the Hadoop FS spec and general mentor of the S3A work, even when not actively working on it. Been full time on S3A, using Spark as the integration test suite, since March
• #4: Simple goal. Make ASF hadoop at home in cloud infra. It's always been a bit of a mixed bag, and there's a lot with agility we need to address: things fail differently. Step 1: Azure. That's the work with Microsoft on wasb://; you can use Azure as a drop-in replacement for HDFS in Azure Step 2: EMR. More specifically, have the ASF Hadoop codebase get higher numbers than EMR
• #5: This is one of the simplest deployments in cloud: scheduled/dynamic ETL. Incoming data sources saving to an object store; spark cluster brought up for ETL. Either direct cleanup/filter or multistep operations, but either way: an ETL pipeline. HDFS on the VMs for transient storage, the object store used as the destination for data —now in a more efficient format such as ORC or Parquet
• #6: Notebooks on demand. ; it talks to spark in cloud which then does the work against external and internal data; Your notebook itself can be saved to the object store, for persistence and sharing.
• #7: Example: streaming on Azure
• #12: Everything usies the Hadoop APIs to talk to both HDFS, Hadoop Compatible Filesystems and object stores; the Hadoop FS API. There's actually two: the one with a clean split between client side and "driver side", and the older one which is a direct connect. Most use the latter and actually, in terms of opportunities for object store integration tweaking, this is actually the one where can innovate with the most easily. That is: there's nothing in the way. Under the FS API go filesystems and object stores. HDFS is "real" filesystem; WASB/Azure close enough. What is "real?". Best test: can support HBase.
• #14: you used to have to disable summary data in the spark context Hadoop options, but https://issues.apache.org/jira/browse/SPARK-15719 fixed that for you
• #18: This is the history
• #20: Here's a flamegraph of LLAP (single node) with AWS+HDC for a set of TPC-DS queries at 200 GB scale; we should stick this up online only about 2% of time (optimised code) is doing S3 IO. Something at start partitioning data
• #22: Why so much seeking? It's the default implementation of read
• #24: A big killer turned out to be the fact that if we had to break and re-open the connection, on a large file this would be done by closing the TCP connection and opening a new one. The fix: ask for data in smaller blocks; the max of (requested-length, min-request-len). Result: significantly lower cost back-in-file seeking and in very-long-distance forward seeks, at the expense of an increased cost in end-to-reads of a file (gzip, csv). It's an experimental option for this reason; I think I'd like to make it an API call that libs like parquet & orc can explicitly request on their IO: it should apply to all blobstores
• #25: If you look at what we've done, much of it (credit to Rajesh & Chris) is just minimising HTTP requests. Each one can take hundreds of millis, sometimes even seconds due to load balancer issues (tip: reduce DNS TTL on your clients to <30s). A lot of the work internal to S3A was culling those getFileStatus() calls by (a) caching results, (b) not bothering to see if they are needed. Example: cheaper to issue a DELETE listing all parent paths than actually looking to see if they exist, wait for the response, and then delete them. The one at the end, HADOOP-13208, replaces a slow recursive tree walk (many status, many list) with a flat listing of all objects in a tree. This works only for the listStatus(path, true) call —to be benefit you need to use that API call, not do your own treewalk.
• #28: Don't run off saying "hey, 2x speedup". I'm confident we got HDP faster, EMR is still something we'd need to look at more. Data layout is still a major problem here; I think we are still understanding the implications of sharding and throttling. What we do know is that deep/shallow trees are pathological for recursive treewalks, and they end up storing data in the same s3 nodes, so throttling adjacent requests.
• #29: Disclaimer: benchmarks, etc. Data was 200GB TCP-DS stored in s3, workload against same cluster.
• #30: And the result. Yes, currently we are faster in these benchmarks. Does that match to the outside world? If you use ORC & HIve, you will gain from the work we've done. There are still things which are pathologically bad, especially deep directory trees with few files
• #35: This invariably ends up reaching us on JIRA, to the extent I've got a document somewhere explaining the problem in detail. It was taken away because it can corrupt your data, without you noticiing. This is generally considered harmful.
• #36: see: Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency for details, essentially it has the semantics HBase needs, that being our real compatibility test.