Константин Макарычев (Sofware Engineer): ИСПОЛЬЗОВАНИЕ SPARK ДЛЯ МАШИННОГО ОБУЧЕНИЯ
Download as PPTX, PDF0 likes52 views
The document discusses implementing machine learning in Apache Spark, highlighting features like preprocessing, model training, and serving pipelines. It specifically showcases the use of logistic regression and the integration of various components such as tokenization and feature hashing. Additionally, it outlines the operational aspects of deploying a trained model using Spark's capabilities.
![apache spark 1
hadoop mapreduce 0
spark machine learning 1
tokenizer
[apache, spark] 1
[hadoop, mapreduce] 0
[spark, machine, learning] 1](https://image.slidesharecdn.com/secon2017-16x9-170427103333/85/Sofware-Engineer-SPARK-8-320.jpg)
![hashing tf
[apache, spark] 1
[hadoop, mapreduce] 0
[spark, machine, learning] 1
[105, 495], [1.0, 1.0] 1
[6, 638, 655], [1.0, 1.0, 1.0] 0
[105, 72, 852], [1.0, 1.0, 1.0] 1](https://image.slidesharecdn.com/secon2017-16x9-170427103333/85/Sofware-Engineer-SPARK-9-320.jpg)
![logistic regression
[105, 495], [1.0, 1.0] 1
[6, 638, 655], [1.0, 1.0, 1.0] 0
[105, 72, 852], [1.0, 1.0, 1.0] 1
0 72 -2.7138781446090308
0 94 0.9042505436914775
0 105 3.0835670890496645
…
0 495 3.2071722417080766
0 722 0.9042505436914775](https://image.slidesharecdn.com/secon2017-16x9-170427103333/85/Sofware-Engineer-SPARK-10-320.jpg)
Константин Макарычев (Sofware Engineer): ИСПОЛЬЗОВАНИЕ SPARK ДЛЯ МАШИННОГО ОБУЧЕНИЯ
- 1. MACHINE LEARNING IN SPARK Константин Макарычев secon 2017
- 2. Big Data: Volume, Velocity, Variety
- 4. val wordCounts = textFile .flatMap(line => line.split(" ")) .map(word => (word, 1)) .reduceByKey((a, b) => a + b) executor executor executor executor executor
- 5. SQL, Streaming, GraphX, MLlib
- 6. Machine Learning: training + serving
- 7. preprocess preprocess train model pipeline
- 8. apache spark 1 hadoop mapreduce 0 spark machine learning 1 tokenizer [apache, spark] 1 [hadoop, mapreduce] 0 [spark, machine, learning] 1
- 9. hashing tf [apache, spark] 1 [hadoop, mapreduce] 0 [spark, machine, learning] 1 [105, 495], [1.0, 1.0] 1 [6, 638, 655], [1.0, 1.0, 1.0] 0 [105, 72, 852], [1.0, 1.0, 1.0] 1
- 10. logistic regression [105, 495], [1.0, 1.0] 1 [6, 638, 655], [1.0, 1.0, 1.0] 0 [105, 72, 852], [1.0, 1.0, 1.0] 1 0 72 -2.7138781446090308 0 94 0.9042505436914775 0 105 3.0835670890496645 … 0 495 3.2071722417080766 0 722 0.9042505436914775
- 11. val tokenizer = new Tokenizer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001) val pipeline = new Pipeline() .setStages(Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(training) model.write.save("/tmp/spark-model")
- 13. val test = spark.createDataFrame(Seq( ("spark hadoop"), ("hadoop learning") )).toDF("text") val model = PipelineModel.load("/tmp/spark-model") model.transform(test).collect()