Spark Streamingによるリアルタイムユーザ属性推定
5 likes1,910 views
Spark Streaming allows processing of live data streams using Spark. It works by dividing the data stream into batches called micro-batches, which are then processed using Spark's batch engine to generate RDDs. This allows for fault tolerance, exactly-once processing, and integration with other Spark APIs like MLlib and GraphX.
![Micro-batch
• RDD HBase
dstream.foreachRDD { rdd =>
val hbaseConf = createHbaseConfiguration()
val jobConf = new Configuration(hbaseConf)
jobConf.set("mapreduce.job.output.key.class", classOf[Text].getName)
jobConf.set("mapreduce.job.output.value.class", classOf[Text].getName)
jobConf.set("mapreduce.outputformat.class",
classOf[TableOutputFormat[Text]].getName)
new PairRDDFunctions(rdd.map(hbaseConvert)).saveAsNewAPIHadoopDataset(jobConf)
}
// RDD[(String, Map[K,V])] RDD[(String, Put)]
def hbaseConvert(t:(String, Map[String, String])) = {
val p = new Put(Bytes.toBytes(t._1))
t._2.toSeq.foreach(
m => p.addColumn(Bytes.toBytes("seg"),
Bytes.toBytes(m._1), Bytes.toBytes(m._2))
)
(t._1, p)
}
19
0.5 1](https://image.slidesharecdn.com/20151209sparkstreaming-160917053325/85/Spark-Streaming-19-320.jpg)
![Spark Streaming :
• : →
RDD → RDD DStream → DStream
• 1Micro-batch
23
// RDD → RDD
val input:RDD[String] = sparkContext.makeRDD(Seq("a", "b", “c"))
// DStream → DStream
val queue = scala.collection.mutable.Queue(rdd)
val dstream:DStream[String] =
sparkStreamingContext.queueStream(queue)](https://image.slidesharecdn.com/20151209sparkstreaming-160917053325/85/Spark-Streaming-23-320.jpg)
}
}
24](https://image.slidesharecdn.com/20151209sparkstreaming-160917053325/85/Spark-Streaming-24-320.jpg)
: JavaRDD[T] =
new JavaRDD[T](rdd)
implicit def toRDD[T](rdd: JavaRDD[T]): RDD[T] = rdd.rdd
}](https://image.slidesharecdn.com/20151209sparkstreaming-160917053325/85/Spark-Streaming-26-320.jpg)
Spark Streamingによるリアルタイムユーザ属性推定
- 2. • • Spark Streaming • • • Spark Streaming Tips • 2
- 3. : / SAEKI Yoshiyasu : IT : Web 4 9 R&D Hadoop, Kafka, Storm, Spark, Druid : RICOH Theta ( ) + Google Cardboard 3
- 5. 5
- 7. • • ≒ … • • ! OS etc. 7
- 8. 1. Web (JavaScript) 2. fluentd Kafka 8
- 9. : fluentd → Kafka • fluent-plugin-kafka • https://github.com/htgc/fluent-plugin-kafka • output type = kafka_buffered (on file) • Kafka 0.8.2.2 • 0.9.0 • ACL 9
- 10. 10
- 11. Suro • Netflix • https://github.com/Netflix/suro • : Kafka Consumer API Thrift API • : • HDFS • AWS S3 • Kafka Producer • Elasticsearch • 11 LinkedIn Gobblin
- 12. Hadoop • • HDFS • MLlib • Streaming linear regression (Classification) • Streaming k-means (Clustering) • 12
- 14. Kafka • Direct Approach (>= Spark 1.3) • • Exactly-once • Kafka Simple Consumer API Direct Approach 14
- 15. Spark Streaming 1 15 http://spark.apache.org/docs/1.5.2/streaming-programming-guide.html RDD @ time1 RDD @ time2 RDD @ time3 RDD @ time4
- 19. Micro-batch • RDD HBase dstream.foreachRDD { rdd => val hbaseConf = createHbaseConfiguration() val jobConf = new Configuration(hbaseConf) jobConf.set("mapreduce.job.output.key.class", classOf[Text].getName) jobConf.set("mapreduce.job.output.value.class", classOf[Text].getName) jobConf.set("mapreduce.outputformat.class", classOf[TableOutputFormat[Text]].getName) new PairRDDFunctions(rdd.map(hbaseConvert)).saveAsNewAPIHadoopDataset(jobConf) } // RDD[(String, Map[K,V])] RDD[(String, Put)] def hbaseConvert(t:(String, Map[String, String])) = { val p = new Put(Bytes.toBytes(t._1)) t._2.toSeq.foreach( m => p.addColumn(Bytes.toBytes("seg"), Bytes.toBytes(m._1), Bytes.toBytes(m._2)) ) (t._1, p) } 19 0.5 1
- 20. 20
- 21. Spark Streaming : • DStream RDD • Spark Spark Streaming 21 http://spark.apache.org/docs/1.5.2/streaming-programming-guide.html
- 22. Spark Streaming : • Fault Tolerance • Micro-batch • YARN • YARN Dynamic Resource Allocation • 22
- 23. Spark Streaming : • : → RDD → RDD DStream → DStream • 1Micro-batch 23 // RDD → RDD val input:RDD[String] = sparkContext.makeRDD(Seq("a", "b", “c")) // DStream → DStream val queue = scala.collection.mutable.Queue(rdd) val dstream:DStream[String] = sparkStreamingContext.queueStream(queue)
- 24. Spark Streaming : • spark-testing-base • https://github.com/holdenk/spark-testing-base class JsonElementCountTest extends StreamingSuiteBase { test("simple") { val input = List(List("aa"), List("bb")) val expected = List(List("AA"), List(“BB")) testOperation[String, String]( input, converterMethod _, expected, useSet = true) } } 24
- 25. Spark Streaming : • Window-based micro-batch • • o.a.spark.streaming.util.ManualClock • private class Scala • http://mkuthan.github.io/blog/2015/03/01/spark- unit-testing/ 25
- 26. Spark Streaming : • Scala Java • • Spark Streaming Kafka HBase Scala • Java 26 // api/java/JavaRDD.scala object JavaRDD { implicit def fromRDD[T: ClassTag](rdd: RDD[T]): JavaRDD[T] = new JavaRDD[T](rdd) implicit def toRDD[T](rdd: JavaRDD[T]): RDD[T] = rdd.rdd }
- 27. 27 • • • = • Spark Streaming • MLlib • GraphX