Machine learning with Apache Spark on Kubernetes | DevNation Tech Talk
2 likes3,047 views
The document discusses machine learning implementation using Apache Spark on Kubernetes, focusing on tasks such as configuration, data collection, feature extraction, and model management. It covers aspects of the machine learning workflow, including model training, validation, and deployment, alongside monitoring and resource management. Additionally, it highlights the use of Spark Structured Streaming for processing real-time data and performing aggregations.
Machine learning with Apache Spark on Kubernetes | DevNation Tech Talk
- 1. Machine Learning With Apache Spark On Kubernetes Erik Erlandson, Red Hat, Inc. eje@redhat.com @ManyAngled
- 2. configuration data collection feature extraction process management analysis tools monitoring serving infrastructure machine resource management data verification (Adapted from Sculley et al., “Hidden Technical Debt in Machine Learning Systems.” NIPS 2015) ML In An Application Context
- 3. configuration data collection feature extraction process management analysis tools monitoring serving infrastructure machine resource management data verification (Adapted from Sculley et al., “Hidden Technical Debt in Machine Learning Systems.” NIPS 2015) ML In An Application Context
- 4. configuration data collection feature extraction process management analysis tools monitoring serving infrastructure machine resource management data verification (Adapted from Sculley et al., “Hidden Technical Debt in Machine Learning Systems.” NIPS 2015) ML In An Application Context
- 5. configuration data collection feature extraction process management analysis tools monitoring serving infrastructure machine resource management data verification (Adapted from Sculley et al., “Hidden Technical Debt in Machine Learning Systems.” NIPS 2015) ML In An Application Context
- 6. feature engineering model training and tuning data collection and cleaning model validation model deployment monitoring, validation codifying problem and metrics codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation ML Workflow
- 7. feature engineering model training and tuning data collection and cleaning model validation model deployment monitoring, validation codifying problem and metrics codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation ML Workflow
- 8. feature engineering model training and tuning data collection and cleaning model validation model deployment monitoring, validation codifying problem and metrics codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation ML Workflow
- 9. Temporibus quis nihil vel consequatur. Veniam rerum sapiente qui sunt pariatur sit deleniti veniam. Itaque quae est nulla necessitatibus qui voluptate assumenda. Earum et sapiente voluptatem et et. Unde tempora temporibus molestias occaecati labore rem. Facilis aut at fuga eaque ipsam. Nulla dolorem impedit nulla dolor non rerum vel. Quia quo dolores natus fuga ipsum. Eum ea ad laboriosam nemo corporis vel. feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning data collection and cleaning codifying problem and metrics ML Workflow
- 10. feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics feature engineering data collection and cleaning f( ) = 0.67 0.57 0.84 0.08 0.42 0.01 ML Workflow
- 11. data scientists application developersdata engineers federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
- 12. data scientists application developersdata engineers federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
- 13. data scientists application developersdata engineers federate trainmodels events databases file, object storage management web and mobile reporting developer UItransform transform transform archive
- 14. Spark’s Compute Model 0 1 2 Executor 3 4 5 Executor 6 7 8 ExecutorDriver 0 1 2 3 4 5 6 7 8App Logical View Physical View
- 15. Spark’s Compute Model 0 1 2 ExecutorDriver 3 4 5 Executor 6 7 8 Executor
- 16. Spark’s Compute Model 0 1 2 ExecutorDriver 3 4 5 Executor 6 7 8 Executor λx: x * 2
- 17. Spark’s Compute Model 0 1 2 ExecutorDriver 3 4 5 Executor 6 7 8 Executor λx: x * 2 λx: x * 2 λx: x * 2 λx: x * 2
- 18. Spark’s Compute Model 0 2 4 ExecutorDriver 12 14 16 Executor λx: x * 2 λx: x * 2 λx: x * 2 6 8 10 Executor λx: x * 2
- 19. Spark on Kubernetes 0 2 4 ExecutorDriver 12 14 16 Executor λx: x * 2 λx: x * 2 λx: x * 2 6 8 10 Executor λx: x * 2 Driver Pod Executor Pod Executor Pod Executor Pod
- 20. Spark Structured Streaming records.show(5) +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ 20 records.groupBy($"user_id") .agg(avg($"wordcount").alias("avg")) .orderBy($"avg".desc) .show(5) +----------+----+ | user_id| avg| +----------+----+ |9438801796|42.0| |0837938601|41.0| |0004926696|40.0| |7439949213|39.0| |2505585758|39.0| +----------+----+
- 21. Spark Structured Streaming 21 val r = records.groupBy($"user_id") .agg(avg($"wordcount").alias("avg")) .orderBy($"avg".desc) val query = r.writeStream //… +----------+----+ | user_id| avg| +----------+----+ |9438801796|42.0| |0837938601|41.0| |0004926696|40.0| |7439949213|39.0| |2505585758|39.0| +----------+----+ val query = records .writeStream //… +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+
- 22. Spark Structured Streaming 22 val r = records.groupBy($"user_id") .agg(avg($"wordcount").alias("avg")) .orderBy($"avg".desc) val query = r.writeStream //… +----------+----+ | user_id| avg| +----------+----+ |9438801796|42.0| |0837938601|41.0| |0004926696|40.0| |7439949213|39.0| |2505585758|39.0| +----------+----+ val query = records .writeStream //… +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+
- 23. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+
- 24. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ CSV
- 25. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL DB CSV
- 26. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL DB CSV Kafka
- 27. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL DB CSV Kafka ...
- 28. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL SQL DB CSV Kafka ...
- 29. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL DataFrame API SQL DB CSV Kafka ...
- 30. Spark Structured Streaming DataFrame +----------+---------+ | user_id|wordcount| +----------+---------+ |6458791872| 12| |7699035787| 5| |2509155359| 9| |9914782373| 18| |7816616846| 12| +----------+---------+ SQL DataFrame API SQL DB CSV Kafka ...User Defined
- 31. Open Data Hub
- 32. Open Data Hub