[FrontDays'2017] Леонид Блохин (Big Data Engineer): Мист. Сервис для работы с Apache Spark
Download as PPTX, PDF0 likes129 views
This document discusses Mist, an open source platform for exposing Apache Spark jobs through REST APIs. Mist allows users to work with predictive services through a higher level of abstraction without needing to manage low-level Spark configurations. It supports Spark SQL, MLlib and GraphX jobs written in Scala or Python. Mist handles Spark context orchestration, provides real-time low latency model serving, and supports recovery of jobs after failures. The document demonstrates how to configure Mist and develop Spark jobs to run on the platform through REST or MQTT APIs. Future plans include adding support for Apache Kafka streaming and AMQP.
![Mist
www.provectus.com
Configuration
21
mist.context-defaults.spark-conf = {
spark.master = local[*]
spark.default.parallelism = 128
spark.driver.memory = "10g"
spark.scheduler.mode = "FAIR"
}
mist.http.on = true
mist.http.host = "192.168.10.13"
mist.http.port = 2003](https://image.slidesharecdn.com/frontdaysmist-170522085210/85/FrontDays-2017-Big-Data-Engineer-Apache-Spark-21-320.jpg)
![Mist
www.provectus.com
Configuration
25
mist.context.<namespace>.streaming-duration = 1 sec
mist.context-settings.onstart = [“foo”,”bar”]
mist.http.router-config-path = ./config/router.conf](https://image.slidesharecdn.com/frontdaysmist-170522085210/85/FrontDays-2017-Big-Data-Engineer-Apache-Spark-25-320.jpg)
![Mist
Spark Job at Mist
libraryDependencies += "io.hydrosphere" %% "mist-lib-spark2" % "0.11.0"
import io.hydrosphere.mist.lib.spark2._
object MyCoolMistJob extends MistJob {
def execute(): Map[String, Any] = {
val rdd = context.parallelize()
...
return result.asInstance[Map[String, Any]]
}
}
www.provectus.com
29](https://image.slidesharecdn.com/frontdaysmist-170522085210/85/FrontDays-2017-Big-Data-Engineer-Apache-Spark-29-320.jpg)
![Mist
Spark Job at Mist
libraryDependencies += "io.hydrosphere" %% "mist-lib-spark2" % "0.11.0"
import io.hydrosphere.mist.lib.spark2._
import io.hydrosphere.mist.lib.spark2.ml._
object MyCoolMistJob extends MistJob {
def train(savePath: String): Map[String, Any] = {...}
def serve(modelPath: String, features: List[Array[Double]]): Map[String, Any] = {...}
}
www.provectus.com
30](https://image.slidesharecdn.com/frontdaysmist-170522085210/85/FrontDays-2017-Big-Data-Engineer-Apache-Spark-30-320.jpg)
[FrontDays'2017] Леонид Блохин (Big Data Engineer): Мист. Сервис для работы с Apache Spark
- 1. 1 Леонид Блохин Mist Apache Spark analytics as a service
- 2. 2 В ДЕТСТВЕ БЫЛ БЛАГОВОСПИТАННЫМ МАЛЬЧИКОМ, НО НАЧАЛ ИГРАТЬ В ОРЛЯНКУ, СВЯЗАЛСЯ С ПИРАТАМИ И ПОКАТИЛОСЬ... BIG DATA ENGINEER, ХАРАКТЕР НЕИСТОВЫЙ. ЖЕНАТ. ЛЕОНИД БЛОХИН www.provectus.com 2
- 3. Содержание доклада • Apache Spark • Hydrosphere Mist • FrontEnd and BigData www.provectus.com 3
- 7. Resilient Distributed Dataset Распределенный набор данных, который знает как себя вычислить. RDD может быть лениво вычисляемой при запросе может быть и материализована — распределенно, в памяти или на диске (или в памяти с вытеснением на диск) разбита на партиции — это минимальный объем RDD, который будет обработан каждым рабочим узлом. www.provectus.com 7
- 11. Spark MLlib ● Classification: logistic regression, naive Bayes,... ● Regression: generalized linear regression, isotonic regression,... ● Decision trees, random forests, and gradient-boosted trees ● Recommendation: alternating least squares (ALS) ● Clustering: K-means, Gaussian mixtures (GMMs),... ● Topic modeling: latent Dirichlet allocation (LDA) ● Feature transformations: standardization, normalization, hashing,... ● Model evaluation and hyper-parameter tuning ● ML Pipeline construction ● ML persistence: saving and loading models and Pipelines ● Survival analysis: accelerated failure time model ● Frequent itemset and sequential pattern mining: FP-growth, association rules, PrefixSpan ● Distributed linear algebra: singular value decomposition (SVD), principal component analysis (PCA),... ● Statistics: summary statistics, hypothesis testing,... www.provectus.com
- 12. Spark GraphX ● PageRank ● Connected components ● Label propagation ● SVD++ ● Strongly connected components ● Triangle count 12 www.provectus.com 12
- 13. 13
- 14. Mist www.provectus.com 14 http://hydrosphere.io/ Hydrosphere – Opensource Big Data and Analytics platform with DevOps culture in mind.
- 15. Mist www.provectus.com 15 ANALYTICS AS A SERVICE Make your applications smarter. Routine CRUD applications should be deprecated. Add predictive and interactive features into your product to improve the engagement with your users. Tap into big data analytics as a service. No more shared databases and low- level APIs. Hydrosphere.io allows you to work with predictive services on a right degree of abstraction and focus on customer happiness and usability. Get educated from data scientists. Working in a squad with data scientists and big data engineers facilitates knowledge sharing and exposure to new technologies.
- 17. Mist Exposing Apache Spark jobs through REST API Spark 2.1.0 support! HTTP & Messaging (MQTT) API Scala and Python Spark jobs support Support for Spark SQL and Hive High Availability and Fault Tolerance Self Healing after driver program failure Powerful logging Clear end-user API www.provectus.com 17 Why We Needed a Mist
- 18. Mist www.provectus.com 18 Spark Contexts orchestration - Cluster of Sark Clusters: manages multiple Spark contexts in separate JVMs or Dockers
- 19. Mist www.provectus.com 19 Realtime low latency models serving/scoring
- 20. Mist docker run -p 2003:2003 -v /var/run/docker.sock:/var/run/docker.sock -d hydrosphere/mist:master-2.1.0 mist www.provectus.com 20 Running
- 21. Mist www.provectus.com Configuration 21 mist.context-defaults.spark-conf = { spark.master = local[*] spark.default.parallelism = 128 spark.driver.memory = "10g" spark.scheduler.mode = "FAIR" } mist.http.on = true mist.http.host = "192.168.10.13" mist.http.port = 2003
- 22. Mist www.provectus.com Configuration 22 mist.mqtt.on = true mist.mqtt.host = "192.168.10.33" mist.mqtt.port = 1883 # mist listens this topic for incoming requests mist.mqtt.subscribeTopic = "foo" # mist answers in this topic with the results mist.mqtt.publishTopic = "foo"
- 23. Mist www.provectus.com Configuration 23 # recovery job (off by default) mist.recovery.on = true mist.recovery.multilimit = 10 mist.recovery.typedb = "MapDb" mist.recovery.dbfilename = "file.db"
- 24. Mist www.provectus.com Configuration 24 # default settings for all contexts mist.context-defaults # uniq settings mist.context.<namespace> # timeout for each job in context mist.contextDefaults.timeout = 100 days # mist can kill context after job finished (off by default) mist.contextDefaults.disposable = false mist.context.<namespace>.worker-downtime = Inf
- 25. Mist www.provectus.com Configuration 25 mist.context.<namespace>.streaming-duration = 1 sec mist.context-settings.onstart = [“foo”,”bar”] mist.http.router-config-path = ./config/router.conf
- 26. Mist www.provectus.com Mist RESTificated Routes 26 forecast = { path = 'hdfs://hdfs-host/jobs/forecast-job.jar', className = 'FastForecast$', namespace = 'production-namespace' }
- 28. Mist www.provectus.com Mist RESTificated Routes 28 HTTP POST /api/forecast { "param": 1 } MQTT { "route": "forecast", "parameters": { "param": 1 } }
- 29. Mist Spark Job at Mist libraryDependencies += "io.hydrosphere" %% "mist-lib-spark2" % "0.11.0" import io.hydrosphere.mist.lib.spark2._ object MyCoolMistJob extends MistJob { def execute(): Map[String, Any] = { val rdd = context.parallelize() ... return result.asInstance[Map[String, Any]] } } www.provectus.com 29
- 30. Mist Spark Job at Mist libraryDependencies += "io.hydrosphere" %% "mist-lib-spark2" % "0.11.0" import io.hydrosphere.mist.lib.spark2._ import io.hydrosphere.mist.lib.spark2.ml._ object MyCoolMistJob extends MistJob { def train(savePath: String): Map[String, Any] = {...} def serve(modelPath: String, features: List[Array[Double]]): Map[String, Any] = {...} } www.provectus.com 30
- 32. Mist www.provectus.com 32 HTTP или MQTT POST http://localhost:2003/api/forecast { “points”:{"lat": 56.1241, “lng”: 42.3457,...} “distance”: 254 }
- 36. Mist www.provectus.com 36 Apache Kafka support Bi-directional streaming API AMQP support Your contributions are very welcome on Github! https://github.com/Hydrospheredata/mist Road Map
- 37. Леонид Блохин телефон: 8 (917) 295 - 40 - 49 Email: lblokhin@provectus.com www.provectus.com Благодарю за внимание!