Apache Spark talk @ The Amsterdam Applied Machine Learning meetup group
1 like1,197 views
This talk provides an overview of Apache Spark, a tool for distributed computing. It describes Resilient Distributed Datasets (RDDs) as Spark's core data structure, which are immutable and distributed collections of records across a cluster. RDDs support transformations and actions, where transformations create new RDDs and actions return data or materialize an RDD. The talk highlights Spark's language bindings for Java, Scala, and Python and its ability to work interactively from a shell. It also notes Spark's compatibility with Hadoop and ability to deploy on YARN and read from HDFS.
Related topics:1 of 27
Downloaded 21 times
More Related Content
What's hot (20)
Ad
Similar to Apache Spark talk @ The Amsterdam Applied Machine Learning meetup group (20)
Ad
More from fvanvollenhoven (8)
Recently uploaded (20)
Apache Spark talk @ The Amsterdam Applied Machine Learning meetup group
- 1. GoDataDriven PROUDLY PART OF THE XEBIA GROUP @fzk frisovanvollenhoven@godatadriven.com Apache Spark Friso van Vollenhoven for applied machine learning
- 4. This talk is about tools.
- 14. Resilient Distributed Dataset •Immutable set of records (e.g. tuples) •Distributed across a cluster of workers •Stored in RAM or on disk (partially) •Built through transformations •Automatically rebuilt on failure •Possibly replicated
- 15. Operations •Operate on RDD’s •Create a new RDD •Or materialise RDD and return data •Transformations: map, filter, groupBy, etc. •Actions: count, collect, reduce, save, etc.
- 18. The good parts •Language bindings for Java, Scala and Python •Works interactively from a shell: •Scala + IPython (notebook) •Plays nice with Hadoop •Deploy on top of YARN cluster manager •Read data from HDFS •Hadoop-like fault tolerance
- 27. GoDataDriven We’re hiring / Questions? / Thank you! @fzk frisovanvollenhoven@godatadriven.com Friso van Vollenhoven