![MLFlow
pip install mlflow[extras]
https://www.mlflow.org/docs/latest/tutorials-and-examples/
tutorial.html
Open source tracking and deployment of ML models
Not specific to Spark / MLLib](https://image.slidesharecdn.com/mllibwithmlflow-220606221850-f2a5235c/85/MLlib-with-MLFlow-pdf-8-320.jpg)
MLlib with MLFlow.pdf
- 1. MLlib with MLFlow Michelle Hoogenhout July 17th 2021
- 2. What I’ll cover Use MLlib with Mlflow end-to-end to: ● Prepare data in pyspark for use with MLLib ● Train and evaluate several classifier models ● Log model performance with MLFlow Tracking
- 3. What you’ll need ● Pyspark / Docker ● MLflow https://github.com/michellehoog/mllib-example
- 4. Why pyspark? Enables scalable analysis (without having to know Scala!) Allows distributed processing Creates ML pipelines Interacts with Pandas
- 5. What algorithms are available on pyspark MLLib? ● Variety of classification and regression models, incl. ○ Linear & Logistic Regression ○ Tree-based models ○ Multilayer Perceptron ○ Naive Bayes ● Clustering ● Collaborative filtering ● Frequent pattern mining
- 6. Spark workflow ● DataFrame: Spark ML uses DataFrame from Spark SQL as an ML dataset, which can hold a variety of data types. E.g., a DataFrame could have different columns storing text, feature vectors, true labels, and predictions. ● Transformer: A Transformer is an algorithm which can transform one DataFrame into another DataFrame. E.g., an ML model is a Transformer which transforms DataFrame with features into a DataFrame with predictions. ● Estimator: An Estimator is an algorithm which can be **fit** on a DataFrame to produce a Transformer. E.g., a learning algorithm is an Estimator which trains on a DataFrame and produces a model. ● Pipeline: A Pipeline chains multiple Transformers and Estimators together to specify an ML workflow. ● Parameter:: All Transformers and Estimators now share a common API for specifying parameters.
- 7. Things to note Data format ● Dense format ● Numeric and zero-indexed (non-negative for Naive Bayes) ● Named ‘label’ and ‘features’ Pipelines
- 8. MLFlow pip install mlflow[extras] https://www.mlflow.org/docs/latest/tutorials-and-examples/ tutorial.html Open source tracking and deployment of ML models Not specific to Spark / MLLib
- 9. MLFlow can log: ● Git commit hash ● Start & end time ● Source ● Parameters ● Metrics ● Artifacts (output)
- 11. MLflow tracking overview Step 1. Create experiment Step 2. Add runs to your code Step 3. View logs
- 12. MLflow tracking overview All MLflow runs are logged to the active experiment, which can be set using any of the following ways: ● Use the mlflow.set_experiment() command. ● Use the experiment_id parameter in the mlflow.start_run() command. ● Set one of the MLflow environment variables MLFLOW_EXPERIMENT_NAME or MLFLOW_EXPERIMENT_ID. If no active experiment is set, runs are logged to the notebook experiment.
- 13. Viewing the Tracking MLflow UI The tracking API writes data to local ./mlruns directory. To view: Run MLflow instance with mlflow ui MLflow’s Tracking UI: http://localhost:5000/#/