End-to-End ML pipelines with Beam, Flink, TensorFlow and Hopsworks.

End-to-End ML pipelines with Beam,
Flink, TensorFlow, and Hopsworks
Theoﬁlos Kakantousis
Software Engineer & COO
@theoﬁloskak
3rd Apache Beam meetup, Stockholm, July 2019

Agenda
1. End-to-end ML pipelines
2. What is Hopsworks
3. Beam Portable Runner with Flink in Hopsworks
4. ML Pipelines with Beam and TensorFlow Extended
5. Demo

End-to-end ML Pipeline
Data
Prep
Data
Ingest
Train Serve
Online
Monitor
Distributed Storage
Raw
Data
Data
Lake
Resource Manager

Typical Feature Store pipeline

Hopsworks Timeline
“If you’re working with big data and Hadoop, this one paper could repay your
investment in the Morning Paper many times over.... HopsFS is a huge win.”
- Adrian Colyer, The Morning Paper
World’s first Hadoop
platform to support
GPUs-as-a-Resource
World’s fastest
Hadoop Published
at USENIX FAST
with Oracle and
Spotify
World’s First
Open Source Feature
Store for Machine
Learning
World’s First
Distributed Filesystem to
store small files in metadata
on NVMe disks
Winner of IEEE
Scale Challenge
2017
with HopsFS -
1.2m ops/sec
2017
World’s most scalable
Filesystem with
Multi Data Center
Availability
2018 2019
World’s first
Open Source Platform
to support TensorFlow
Extended (TFX) on
Beam

True Project-based multi-tenancy
Proj-XProject-42
Kafka TopicResources /Projs/My/Data
Project-AllExperimentsModels
Experiments

Hopsworks REST API
● Manage Hopsworks resources via the REST API
○ Projects
○ Datasets
○ Jobs
○ FeatureStore
○ Experiments
○ ModelServing
○ Kafka
○ ...
● Documented with Swagger and hosted on SwaggerHub
○ https://app.swaggerhub.com/apis-docs/logicalclocks/hopsworks-api/0.10.0

Beam Portable Runner
Beam Model: Fn Runners
Apache
Flink
Apache
Spark
Beam Model: Pipeline Construction
Other
LanguagesBeam Java
Beam
Python
Execution Execution
Cloud
Dataﬂo
w
Execution
1. End users: who want to
write pipelines in a
language that’s familiar.
2. SDK writers: who want
to make Beam concepts
available in new
languages.
3. Runner writers: who
have a distributed
processing environment
and want to support
Beam pipelines
https://s.apache.org/apache-beam-project-overview

Beam-as-a-Service in Hopsworks
● Develop Beam pipelines in Python from Jupyter notebooks
● Tooling to simplify deployment and execution
● Manage lifecycle of Beam Portability JobService(JobServer)
● Logging and monitoring of Beam jobs
● SDK Workers(harness) with conda env
● Scalable execution on Flink/Spark clusters

Hopsworks API
● hops-util-py (Python) and HopsUtil(Java)
● Simpliﬁes development:
○ Sets security conﬁg
○ Discover cluster services
○ Helper methods for the Hopsworks REST API
○ ML Experiments
● Manage Beam Runners and Job Service
https://github.com/logicalclocks/hops-util-py/, https://github.com/logicalclocks/hops-util

Beam Portability - Process vs Docker
● Docker:
○ Build image with all your
dependencies
○ Update or modify? build new
containers
○ Additional infrastructure
components
● Process:
○ Install dependencies on all
servers
○ Management of
dependencies?
○ Easy to update and modify
libraries
○ Challenge? Multi-tenancy &
keep servers in sync
● SDK Worker: SDK-provided program responsible for executing user code
● How to manage the user’s dependencies, libraries, … ?

First class Python: Conda in the Cluster
Conda Repo
Hopsworks Cluster
No need to write
Dockerﬁles

Jupyter dashboard in Hopsworks
● Manage notebook
settings from
dashboard

Jupyter dashboard in Hopsworks
● Execute a Beam Python
pipeline
● With the Python kernel
either in a docker
container managed by
Kubernetes or as a local
Python process.
● In a PySpark executor in
the cluster.

Notebooks as Beam jobs in ML pipelines

Beam portability architecture in Hopsworks
https://www.slideshare.net/ThomasWeise/python-streaming-pipelines-on-flink-beam-meetup-at-lyft-2019

HopsFS
Local/YARN/K8s
Hopsworks
Session cluster on YARN

Local/YARN/K8s
Compiled and shipped with
HopsFS dependencies
Hopsworks
HopsFS
Local/YARN/K8s
hops-util.py

Local/YARN/K8s
Hopsworks
HopsFS
Local/YARN/K8s
hops-util.py
# creates and starts runner
# localizes Job Service jar ﬁle
from HopsFS
# Provides arguments (ports,
artifacts_dir, etc.)
# Start Job Service and
returns host,port
# Job Service automatically
shuts down when Python
pipeline shuts down
host,port = start_runner()

Local/YARN/K8s
Hopsworks
HopsFS
Local/YARN/K8s
hops-util.py
Python conda env and
Hopsworks env
variables are set for
SDKWorker

Hopsworks API
https://github.com/logicalclocks/hops-util-py/, https://github.com/logicalclocks/hops-util
def start_runner(
runner="flink",
runner_name="session",
runner_config=config)
def start_jobservice(
runner = "Resources",
artifacts_dir="Resources",
job_server_path="hdfs:///user/flink/",
job_server_jar="beam-runners-flink-1.8-job-server-2.13.0.jar",
sdk_worker_parallelism=1)
hops.beam.start_runner()
hops.beam.start_jobservice()

Logging
● Flink JobManager and TaskManager
● Beam Job service
○ Local mode - logs in project’s Jupyter staging dir
○ Cluster - logs in the PySpark container where process is running.
● SDK Worker
○ Logs are in the Flink TaskManager container
● Collect and visualize with the ELK stack
○ Logs are accessible only by project members

Secure Beam with TLS certiﬁcates

Hidden Technical Debt in Machine Learning Systems
Data validation
Distributed
Training
Model
Serving
A/B
Testing
Monitoring
Pipeline Management
HyperParameter
Tuning
Feature Engineering
Data
Collection
Hardware
Management
Data Model Prediction
φ(x)
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

TensorFlow Extended (TFX)
https://www.tensorflow.org/tfx

TFX on a Flink Cluster with Portable Runner

Distributed Deep Learning in Hopsworks
Executor 1 Executor N
Driver
HopsFS (HDFS)TensorBoard Model Serving

Experiments - TensorBoard
● Repeatable
experiments
● Manage
experiments
metadata
● Integration with
Tensorboard

Apache Airﬂow-as-a-Service
● Airﬂow available as a
multi-tenant service
in a Hopsworks
● Develop pipelines
with Hopsworks
operators and
sensors

Apache Airﬂow-as-a-Service - TFX pipeline
●

Horizontally Scalable ML Pipelines
Raw Data
Event Data
Monitor
HopsFS
Serving
Feature Store /
TFX Transform
Data PrepIngest DeployExperiment /
Train
logs
logs
Metadata Store
External
Model Analysis
FeatureStore

Compatibility...
● Hopsworks-1.0
● Beam 2.13.0
● Flink 1.8.0
● TensorFlow 1.14.0
● TFX 0.13
● TensorFlow Model Analysis 0.13.2

Conclusions & Future Work
● Summary
○ Hopsworks v1.0 the ﬁrst on-prem open source horizontally scalable platform to support Beam
Portable Runner with Flink runner
○ Develop and Manage lifecycle of horizontally scalable End-to-End ML Pipelines with Beam and
TFX
● Future Work
○ Add support for Spark Runner
○ Export metrics for Flink runner to InﬂuxDB and visualize with Grafana

Contributors
Jim Dowling, Seif Haridi, Gautier Berthou, Salman Niazi, Mahmoud Ismail, Theoﬁlos Kakantousis, Ermias
Gebremeskel, Fabio Buso, Antonios Kouzoupis, Kim Hammar, Steffen Grohsschmiedt, Alex Ormenisan,
Robin Andersson, Moritz Meister, Kajetan Maliszewski, Netsanet Gebretsadkan Kidane, Sina Sheikholeslami,
Joel Stenkvist, August Bonds, Vasileios Giannokostas, Johan Svedlund Nordström,Rizvi Hasan, Paul Mälzer,
Bram Leenders, Juan Roca, Misganu Dessalegn, K “Sri” Srijeyanthan, Jude D’Souza, Alberto Lorente, Andre
Moré, Ali Gholami, Davis Jaunzems, Stig Viaene, Hooman Peiro, Evangelos Savvidis, Qi Qi, ...

How to get started with Hopsworks?
@hopsworks
Register for a free account at: www.hops.site
Images available for AWS, GCE, Virtualbox.
https://www.logicalclocks.com/
https://github.com/logicalclocks/hopsworks
https://www.meetup.com/HopsML-Stockholm
Reach us
@logicalclocks

End-to-End ML pipelines with Beam, Flink, TensorFlow and Hopsworks.

More Related Content

What's hot (20)

Similar to End-to-End ML pipelines with Beam, Flink, TensorFlow and Hopsworks. (20)

Recently uploaded (20)

End-to-End ML pipelines with Beam, Flink, TensorFlow and Hopsworks.