Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East talk by Jeff Smith and Rohan Aletty
4 likes1,497 views
This document discusses how Spark can serve as a gateway to typed functional programming by introducing developers to Scala and functional programming concepts. It covers how Spark leverages concepts like lazy evaluation, immutable data structures, higher-order functions, and types to provide a powerful and scalable framework. While complex, these functional techniques help tackle challenges of scale and complexity that arise in real-world AI systems.
![Functions in Scala
def average(observations: List[Double]) {
observations.sum / observations.size
}
average(inputData)](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-19-320.jpg)
![Mutation
visits = {"Church": 2, "Backus": 1, "McCarthy": 4}
old_value = visits["Backus"]
visits["Backus"] = old_value + 1](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-25-320.jpg)
![Spark — Types
case class Person(name: String, age: Int)
val peopleDS = spark.read.json(path).as[Person]
val ageGroupedDs = peopleDS.groupBy(_.age)](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-40-320.jpg)
![Spark — Learning Types
trait Person { def name: String }
case class Student(name: String, grade: String) extends Person
case class Professional(name: String, job: String) extends Person
val personRDD: RDD[Person] = sc.parallelize(…)
// working with both union and product types
val mappedRDD: RDD[String] = personRDD.map {
case Student(name, grade) => grade
case Professional(name, job) => job
}](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-42-320.jpg)
![Spark — Learning Types
val rdd1: RDD[Person] = sc.parallelize(...)
val rdd2: RDD[String] = rdd1.map("name: " + _) // Compilation error!
val rdd3: RDD[String] = rdd2.map("name: " + _.name) // It works!](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-43-320.jpg)
![Scala — Monads!
trait Monad[M[]] {
// constructs a Monad instance from the given value, e.g. List(1)
def apply[T](v: T): M[T]
// effectively lets you transform values within a Monad
def bind[T, U](m: M[T])(fn: (T) => M[U]): M[U]
}](https://image.slidesharecdn.com/3csmithaletty-170214203130/85/Spark-as-the-Gateway-Drug-to-Typed-Functional-Programming-Spark-Summit-East-talk-by-Jeff-Smith-and-Rohan-Aletty-46-320.jpg)
Spark as the Gateway Drug to Typed Functional Programming: Spark Summit East talk by Jeff Smith and Rohan Aletty
- 1. Spark as the Gateway Drug To Typed Functional Programming Jeff Smith Rohan Aletty x.ai
- 2. Real World AI • Scale is increasing • Complexity is increasing • Human brain size is constant
- 6. Datanauts
- 7. Tools
- 8. Scala • Bleeding edge • Real world
- 9. Spark • Incredibly powerful • Easy to use
- 10. Typed Functional Programming • Powerful abstractions • Tough learning curve
- 11. Functions
- 12. Methods • Collection of statements • Might have side effects • On an object
- 13. Methods public class Dataset { private List<Double> observations; private Double average; public Dataset(List<Double> inputData) { observations = inputData; } }
- 14. Methods public class Dataset { public double getAverage() { Double runningSum = 0.0; for (Double observation : observations) { runningSum += observation; } average = runningSum / observations.size(); return average; } }
- 15. Methods public class Dataset { public void setObservations(List<Double> inputData) { observations = inputData; } }
- 16. Methods public class Dataset { private List<Double> observations; private Double average; public Dataset(List<Double> inputData) { observations = inputData; } public double getAverage() { Double runningSum = 0.0; for (Double observation : observations) { runningSum += observation; } average = runningSum / observations.size(); return average; } public void setObservations(List<Double> inputData) { observations = inputData; } }
- 17. Functions • Collection of expressions • Returns a value • Are objects (in Scala) • Can be in-lined
- 18. Functions in Scala val inputData = List(1.0, 2.0, 3.0)
- 19. Functions in Scala def average(observations: List[Double]) { observations.sum / observations.size } average(inputData)
- 20. Functions in Scala def add(x: Double, y: Double) = { x + y } val sum = inputData.foldLeft(0.0)(add) val average = sum / inputData.size
- 21. Functions in Scala val sum = inputData.foldLeft(0.0)(add) val average = sum / inputData.size inputData.foldLeft(0.0)(_ + _) / inputData.size
- 22. Functions in Spark inputData.foldLeft(0.0)(_ + _) / inputData.size val observations = sc.parallelize(inputData) observations.fold(0.0)(_ + _) / observations.count()
- 23. Immutability
- 24. Mutation • Changing an object
- 25. Mutation visits = {"Church": 2, "Backus": 1, "McCarthy": 4} old_value = visits["Backus"] visits["Backus"] = old_value + 1
- 26. Immutability • Never changing objects
- 27. Immutability in Scala val visits = Map("Church" -> 2, "Backus" -> 1, "McCarthy" -> 4) val updatedVisits = visits.updated("Backus", 2)
- 28. Immutability in Spark val manyVisits = sc.parallelize(visits.toSeq) val additionalVisit = sc.parallelize(Seq(("Backus", 1))) val updatedVisits = manyVisits.union(additionalVisit) .aggregateByKey(0)(_ + _, _ + _)
- 29. Recap
- 30. Concepts • Higher-order functions • Anonymous functions • Purity of functions
- 31. Concepts • Currying • Referential transparency • Closures • Resilient Distributed Datasets
- 32. Lazy Evaluation
- 33. Functional Programming — Lazy Evaluation • Delaying evaluation of an expression until a value is needed • Two major advantages of lazy evaluation • Deferring computation allows program only evaluate what is necessary • Changing evaluation scheme into to be more efficient
- 34. Spark — Lazy Evaluation • All transformations are lazy • Their existence added to Spark computation DAG • Example DAGs
- 35. Spark — Lazy Evaluation val rdd1 = sc.parallelize(...) val rdd2 = rdd1.map(...) val rdd3 = rdd1.map(...) val rdd4 = rdd1.map(...) rdd3.take(5)
- 36. Spark — Learning Laziness • Advantage 1: (deferred computation) • draws directly from only evaluating parts of DAG that are necessary when executing an action • Advantage 2: (optimized evaluation scheme) • draws directly from pipelining within Spark stages to make execution more efficient
- 37. Types
- 38. Functional Programming — Type Systems • Mechanism for defining algebraic data types (ADTs) which are useful for program structure • i.e. “let’s group this data together and brand it a new type” • Compile time guarantees of correctness of program • e.g. “no, you cannot add Foo to Bar”
- 39. Spark — Types • RDD’s (typed), Datasets (typed), DataFrames (untyped) • Types provide great schema enforcement on a dataset for preventing unexpected behavior
- 40. Spark — Types case class Person(name: String, age: Int) val peopleDS = spark.read.json(path).as[Person] val ageGroupedDs = peopleDS.groupBy(_.age)
- 41. Spark — Learning Types • Spark through Scala also allows learning of pattern matching • ADTs as both product types and union types • Allows us to reason about code easier • Gives us compile time safety
- 42. Spark — Learning Types trait Person { def name: String } case class Student(name: String, grade: String) extends Person case class Professional(name: String, job: String) extends Person val personRDD: RDD[Person] = sc.parallelize(…) // working with both union and product types val mappedRDD: RDD[String] = personRDD.map { case Student(name, grade) => grade case Professional(name, job) => job }
- 43. Spark — Learning Types val rdd1: RDD[Person] = sc.parallelize(...) val rdd2: RDD[String] = rdd1.map("name: " + _) // Compilation error! val rdd3: RDD[String] = rdd2.map("name: " + _.name) // It works!
- 44. Monads
- 45. Functional Programming — Monads • In category theory: • “a monad in X is just a monoid in the category of endofunctors of X” • In functional programming, refers to a container that can: • Inject a value into the container • Perform operations on values returning a container with new values • Flatten nested containers into a single container
- 46. Scala — Monads! trait Monad[M[]] { // constructs a Monad instance from the given value, e.g. List(1) def apply[T](v: T): M[T] // effectively lets you transform values within a Monad def bind[T, U](m: M[T])(fn: (T) => M[U]): M[U] }
- 47. Scala — Monads! • Many monads in Scala • List, Set, Option, etc. • Powerful line of thinking • Helps code comprehension • Reduces error checking logic (pattern matching!) • Can build further transformations: map(), filter(), foreach(), etc.
- 48. Spark — Learning Monads? • We have many “computation builders” -- (RDD’s, Datasets, DataFrames) • Containers on which transformations can be applied • Similar to monads though not identical • No unit function to wrap constituent values • Cannot lift all types into flatMap function unconstrained
- 49. For Later
- 50. Conclusions • Spark introduces all types of devs to Scala • Scala helps people learn typed functional programming • Typed functional programming improves Spark development
- 52. Use the code ctwsparks17 for 40% off!
- 53. Thank You