Distributed DataFrame (DDF) Simplifying Big Data For The Rest Of Us
3 likes1,743 views
This document discusses Distributed DataFrame (DDF), a new technology that aims to simplify working with big data for data scientists and analysts. DDF provides a table-like abstraction on top of big data that offers native R data frame functionality, real-time scoring abilities, data wrangling and modeling tools, and integration with external machine learning libraries, all while allowing users to work in their preferred programming language like R or Python. The technology is being developed to provide an easy to use interface for big data similar to how SQL and R work for small data.
![class RandomSplitRDD( prev: RDD[Array[Double]], seed: Long, lower: Double,
upper: Double, isTraining: Boolean)
extends RDD[Array[Double]](prev) {
override def getPartitions: Array[Partition] = {
val rg = new Random(seed)
firstParent[Array[Double]].partitions.map(x => new SeededPartition(x,
rg.nextInt))
}
override def getPreferredLocations(split: Partition): Seq[String] =
firstParent[Array[Double]].preferredLocations(split.asInstanceOf[SeededPartitio
n].prev)
override def compute(splitIn: Partition, context: TaskContext):
Iterator[Array[Double]] = {
val split = splitIn.asInstanceOf[SeededPartition]
val rand = new Random(split.seed)
if (isTraining) {
firstParent[Array[Double]].iterator(split.prev, context).filter(x => {
val z = rand.nextDouble;
z < lower || z >= upper
})
} else {
firstParent[Array[Double]].iterator(split.prev, context).filter(x => {
val z = rand.nextDouble;
lower <= z && z < upper
})
}
}
}
def randomSplit[Array[Double]](rdd: RDD[Array[Double]], numSplits: Int,
trainingSize: Double, seed: Long): Iterator[(RDD[Array[Double]],
RDD[Array[Double]])] = {
require(0 < trainingSize && trainingSize < 1)
val rg = new Random(seed)
(1 to numSplits).map(_ => rg.nextInt).map(z =>
(new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, true),
new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, false))).toIterator
}
Can’t It Be
Even
Simpler?
WHY](https://image.slidesharecdn.com/sparkddfr7-140710153126-phpapp01/85/Distributed-DataFrame-DDF-Simplifying-Big-Data-For-The-Rest-Of-Us-7-320.jpg)
![class RandomSplitRDD( prev: RDD[Array[Double]], seed: Long, lower: Double,
upper: Double, isTraining: Boolean)
extends RDD[Array[Double]](prev) {
override def getPartitions: Array[Partition] = {
val rg = new Random(seed)
firstParent[Array[Double]].partitions.map(x => new SeededPartition(x,
rg.nextInt))
}
override def getPreferredLocations(split: Partition): Seq[String] =
firstParent[Array[Double]].preferredLocations(split.asInstanceOf[SeededPartitio
n].prev)
override def compute(splitIn: Partition, context: TaskContext):
Iterator[Array[Double]] = {
val split = splitIn.asInstanceOf[SeededPartition]
val rand = new Random(split.seed)
if (isTraining) {
firstParent[Array[Double]].iterator(split.prev, context).filter(x => {
val z = rand.nextDouble;
z < lower || z >= upper
})
} else {
firstParent[Array[Double]].iterator(split.prev, context).filter(x => {
val z = rand.nextDouble;
lower <= z && z < upper
})
}
}
}
def randomSplit[Array[Double]](rdd: RDD[Array[Double]], numSplits: Int,
trainingSize: Double, seed: Long): Iterator[(RDD[Array[Double]],
RDD[Array[Double]])] = {
require(0 < trainingSize && trainingSize < 1)
val rg = new Random(seed)
(1 to numSplits).map(_ => rg.nextInt).map(z =>
(new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, true),
new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, false))).toIterator
}
table = load(“housePrices”)
table.dropNA
table.train.glm(“price”, “sf,zip,beds,baths”)
What
Happiness
Looks Like](https://image.slidesharecdn.com/sparkddfr7-140710153126-phpapp01/85/Distributed-DataFrame-DDF-Simplifying-Big-Data-For-The-Rest-Of-Us-8-320.jpg)
Distributed DataFrame (DDF) Simplifying Big Data For The Rest Of Us
- 1. Distributed DataFrame (DDF) Simplifying Big Data For The Rest Of Us Christopher Nguyen, PhD Co-Founder & CEO DATA INTELLIGENCE FOR ALL
- 2. 1. Motivation: Problem & Solution 2. DDF Features & Benefits 3. Demo Agenda
- 3. • Former Engineering Director of Google Apps (Google Founders’ Award) • Former Professor and Co-Founder of the Computer Engineering program at HKUST • PhD Stanford, BS U.C. Berkeley Summa cum Laude Christopher Nguyen, PhD Adatao Inc. Co-Founder & CEO
- 7. class RandomSplitRDD( prev: RDD[Array[Double]], seed: Long, lower: Double, upper: Double, isTraining: Boolean) extends RDD[Array[Double]](prev) { override def getPartitions: Array[Partition] = { val rg = new Random(seed) firstParent[Array[Double]].partitions.map(x => new SeededPartition(x, rg.nextInt)) } override def getPreferredLocations(split: Partition): Seq[String] = firstParent[Array[Double]].preferredLocations(split.asInstanceOf[SeededPartitio n].prev) override def compute(splitIn: Partition, context: TaskContext): Iterator[Array[Double]] = { val split = splitIn.asInstanceOf[SeededPartition] val rand = new Random(split.seed) if (isTraining) { firstParent[Array[Double]].iterator(split.prev, context).filter(x => { val z = rand.nextDouble; z < lower || z >= upper }) } else { firstParent[Array[Double]].iterator(split.prev, context).filter(x => { val z = rand.nextDouble; lower <= z && z < upper }) } } } def randomSplit[Array[Double]](rdd: RDD[Array[Double]], numSplits: Int, trainingSize: Double, seed: Long): Iterator[(RDD[Array[Double]], RDD[Array[Double]])] = { require(0 < trainingSize && trainingSize < 1) val rg = new Random(seed) (1 to numSplits).map(_ => rg.nextInt).map(z => (new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, true), new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, false))).toIterator } Can’t It Be Even Simpler? WHY
- 8. class RandomSplitRDD( prev: RDD[Array[Double]], seed: Long, lower: Double, upper: Double, isTraining: Boolean) extends RDD[Array[Double]](prev) { override def getPartitions: Array[Partition] = { val rg = new Random(seed) firstParent[Array[Double]].partitions.map(x => new SeededPartition(x, rg.nextInt)) } override def getPreferredLocations(split: Partition): Seq[String] = firstParent[Array[Double]].preferredLocations(split.asInstanceOf[SeededPartitio n].prev) override def compute(splitIn: Partition, context: TaskContext): Iterator[Array[Double]] = { val split = splitIn.asInstanceOf[SeededPartition] val rand = new Random(split.seed) if (isTraining) { firstParent[Array[Double]].iterator(split.prev, context).filter(x => { val z = rand.nextDouble; z < lower || z >= upper }) } else { firstParent[Array[Double]].iterator(split.prev, context).filter(x => { val z = rand.nextDouble; lower <= z && z < upper }) } } } def randomSplit[Array[Double]](rdd: RDD[Array[Double]], numSplits: Int, trainingSize: Double, seed: Long): Iterator[(RDD[Array[Double]], RDD[Array[Double]])] = { require(0 < trainingSize && trainingSize < 1) val rg = new Random(seed) (1 to numSplits).map(_ => rg.nextInt).map(z => (new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, true), new RandomSplitRDD(rdd, z, 0, 1.0 - trainingSize, false))).toIterator } table = load(“housePrices”) table.dropNA table.train.glm(“price”, “sf,zip,beds,baths”) What Happiness Looks Like
- 10. we begin to dream / find solution to fight this evil
- 11. Design Principles Sophistication Power & SpeedEase & Simplicity
- 12. HDFS Web Browser R-Studio Python Business Analyst Data Scientist Data Engineer API API API DDFClient PAClient PIClient Example:
- 13. CLIENT WORKER WORKER WORKERWORKERMASTER Demo Deployment Diagram Airline Dataset, 12GB, 123M rows
- 15. Table-Like Abstraction on Top of Big Data 1
- 17. 2 Native R Data.frame Experience ddf.getSummary() ddf.getFiveNum() ddf.binning(“distance”, 10) ddf.scale(SCALE.MINMAX)
- 19. Real-Time Scoring manager.loadModel(uri) lm.predict(point) Data Wrangling ddf.dropNA() ddf.transform(“speed =distance/duration”) ddf.lm(0.1, 10) ddf.roc(testddf) Model Building & Validation 3 Focus on Analytics, Not MapReduce
- 21. 4 Seamlessly Integrate with External ML Libraries Data Algorithm Intelligence + =
- 23. 5 Share DDFs Seamlessly & Efficiently Can I see your data? ddf://adatao/airline
- 25. 6Work in Your Preferred Language …even Plain English!
- 26. 1 2 3 4 5 6 Table-Like Abstraction on Top of Big Data Seamlessly Integrate with External ML Libraries Share DDFs Seamlessly & efficiently Native R Data.frame Experience Focus on Analytics, not MR Work in Your Preferred Language
- 27. as DDF is to Big Data SQL+R is to Small Data
- 28. www.adatao.com/ddf DATA INTELLIGENCE FOR ALL We’re Open Sourcing DDF!!! 1. Accept Core Committers 2. Clean up & Prep 3. Open up for public access