Spark Summit East 2017: Apache spark and object stores

Editor's Notes

• #3: Now people may be saying "hang on, these aren't spark developers". Well, I do have some integration patches for spark, but a lot of the integration problems are actually lower down: -filesystem connectors -ORC performance -Hive metastore Rajesh has been doing lots of scale runs and profiling, initially for Hive/Tez, now looking at Spark, including some of the Parquet problems. Chris has done work on HDFS, Azure WASB and most recently S3A Me? Co-author of the Swift connector. author of the Hadoop FS spec and general mentor of the S3A work, even when not actively working on it. Been full time on S3A, using Spark as the integration test suite, since March
• #4: This is one of the simplest deployments in cloud: scheduled/dynamic ETL. Incoming data sources saving to an object store; spark cluster brought up for ETL. Either direct cleanup/filter or multistep operations, but either way: an ETL pipeline. HDFS on the VMs for transient storage, the object store used as the destination for data —now in a more efficient format such as ORC or Parquet
• #5: Notebooks on demand. ; it talks to spark in cloud which then does the work against external and internal data; Your notebook itself can be saved to the object store, for persistence and sharing.
• #6: Example: streaming on Azure
• #11: Everything usies the Hadoop APIs to talk to both HDFS, Hadoop Compatible Filesystems and object stores; the Hadoop FS API. There's actually two: the one with a clean split between client side and "driver side", and the older one which is a direct connect. Most use the latter and actually, in terms of opportunities for object store integration tweaking, this is actually the one where can innovate with the most easily. That is: there's nothing in the way. Under the FS API go filesystems and object stores. HDFS is "real" filesystem; WASB/Azure close enough. What is "real?". Best test: can support HBase.
• #12: This is the history
• #21: you used to have to disable summary data in the spark context Hadoop options, but https://issues.apache.org/jira/browse/SPARK-15719 fixed that for you
• #22: It looks the same, you just need to be as aggressive about minimising IO as you can -push down predicates -only select the columns you want -filter -If you read a lot, write to HDFS then re-use. cache()? I don't know. If you do, filter as much as you can first: columns, predicates, ranges, so that parquet/orc can read as little as it needs to, and RAM use is least.
• #24: without going into the details, here are things you will want for Hadoop 2.8. They are in HDP 2.5, possible in the next CDH release. The first two boost input by reducing the cost of seeking, which is expensive as it breaks then re-opens the HTTPS connection. Readahead means that hundreds of KB can be skipped before that connect (yes, it can take that long to reconnect). The experimental fadvise random feature speeds up backward reads at the expense of pure-forward file reads. It is significantly faster for reading in optimized binary formats like ORC and Parquet The last one is a successor to fast upload in Hadoop 2.7. That buffers on heap and needs careful tuning; its memory needs conflict with RDD caching. The new version defaults to buffering as files on local disk, so won't run out of memory. Offers the potential of significantly more effective use of bandwidth; the resulting partitioned files may also offer higher read perf. (No data there, just hearsay).
• #26: This invariably ends up reaching us on JIRA, to the extent I've got a document somewhere explaining the problem in detail. It was taken away because it can corrupt your data, without you noticiing. This is generally considered harmful.
• #27: if your distributor didn't stick the JARs in, you can add the hadoop-aws and hadoop-azure dependencies in the interpreter config credentials: keep out of notebooks. Zeppelin can list its settings too; always dangerous (mind you, so does HDFS and YARN, so an XInclude is handy there) when running in EC2, S3 credentials are now automatically picked up. And, if zeppelin is launched with the AWS env vars set, its invocation of spark-submit should pass them down.
• #28: see: Windows Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency for details, essentially it has the semantics HBase needs, that being our real compatibility test.
• #29: Azure storage is unique in that there's a pubished paper (+ video) on its internals. Well worth looking at to understand what's going on. In contrast, if you want to know S3 internals, well, you can ply the original author with gin and he still won't reveal anything. ADL adds okhttp for HTTP/2 performance; yet another json parser for unknown reasons
• #30: Azure storage is unique in that there's a pubished paper (+ video) on its internals. Well worth looking at to understand what's going on. In contrast, if you want to know S3 internals, well, you can ply the original author with gin and he still won't reveal anything. ADL adds okhttp for HTTP/2 performance; yet another json parser for unknown reasons
• #41: Hadoop 2.8 adds a lot of control here. (credit: Netfllx, + later us & cloudera) -You can define a list of credential providers to use; the default is simple, env, instance, but you can add temporary and anonymous, choose which are unsupported, etc. -passwords/secrets can be encrypted in hadoop credential files stored locally, HDFS -IAM auth is what EC2 VMs need
• #42: And this is the big one, as it spans the lot: Hadoop's own code (so far: distcp), Spark, Hive, Flink, related tooling. If we can't speed up the object stores, we can tune the apps