ODSC East 2017 - Reproducible Research at Scale with Apache Zeppelin and Spark
- 1. REPRODUCIBLE RESEARCH AT SCALE WITH APACHE SPARK AND ZEPPELIN NOTEBOOK CAROLYN DUBY SOLUTIONS ENGINEER, NORTHEAST HORTONWORKS @ODSC OPEN DATA SCIENCE CONFERENCE Boston | May 3-5th
- 2. ABOUT CAROLYN DUBY • Big Data Solutions Architect • High performance data intensive systems • Data science • ScB ScM Computer Science, Brown University • LinkedIn: https://www.linkedin.com/in/carolynduby/ • Twitter: @carolynduby Github: carolynduby • Hortonworks • Innovation through data • Enterprise ready, 100% open source, modern data platforms • Engineering, Technical Support, Professional Services, Training
- 4. AGENDA • What is Reproducible Research? Why do it? • What does it take to do Reproducible Research at scale? • Example with Apache Zeppelin and Spark
- 5. REPRODUCIBLE RESEARCH • Complete details of data analysis methods yielding conclusions • Replication of research on independently collected data • Gold standard
- 6. BENEFITS • Individual productivity • Effective peer review • Answer questions more quickly • Correct errors • Apply methods to other experiments • Increased quality and respect for results • Justify business decisions
- 7. CHALLENGES • Large data sets • Complex analysis • Data lineage • Streaming data • Limited space in publications
- 8. HOW TO DO REPRODUCIBLE RESEARCH • Define Platform • Record all versions of analysis software and installation procedures • Analyze Data • Record all commands to acquire, clean, organize, analyze • Store intermediate results • Version control • Share Methods and Results • Publish • Share full details
- 9. REPRODUCIBLE RESEARCH WITH APACHE OPEN SOURCE • Apache Spark version 2.1 • Cleaning and analysis of large data sets • http://spark.apache.org • Apache Zeppelin Notebook 0.7.0 • Capture automated commands • Visualize data for exploration and results • https://zeppelin.apache.org
- 10. APACHE SPARK • Distributed processing efficiently crunches large data sets • Optimized • Horizontally scalable with multi tenancy • Fault tolerant • One platform for streaming, cleaning, analyzing • Elegant APIs – Scala, Python, Java, R • Many data source connectors – file system, HDFS, Hive, Phoenix, S3, etc
- 11. SPARK LIBRARIES • Same API for all data sources • SQL - http://spark.apache.org/sql/ • Access structured data and combine with other sources • MLLIB - http://spark.apache.org/mllib/ • Machine learning for training models and predicting • GraphX - http://spark.apache.org/graphx/ • Connectivity algorithms • Streaming - http://spark.apache.org/streaming/ • Complex event processing and data ingest
- 12. ZEPPELIN • Notebook • Combine mark down, shell, spark, sql commands in same notebook • Easily integrate with Spark in different languages • Visualize data using graphs and pivot charts • Share notebooks or paragraphs
- 14. ARCHITECTURE Spark Driver Zeppelin Spark Application Master YARN container Spark Executor YARN container Task Task Spark Executor YARN container Task Task Spark Executor YARN container Task Task Client Browser
- 15. GETTING STARTED • Use a distribution • Curated set of compatible open source projects • Sandbox - single node cluster in VM or Azure • https://hortonworks.com/products/sandbox/ • Hortonworks Community Connection • http://community.hortonworks.com • On premise • Use Apache Ambari to manage on premise physical hardware • Cloud • Automated provisioning with Cloudbreak (https://github.com/sequenceiq/cloudbreak) • AWS, Azure, Google Cloud
- 16. ZEPPELIN BASICS • Notes are composed of paragraphs • Paragraph contains code or markdown • Specify interpreter - % <interpreter name> or blank for default • Enter commands • Click play button to run code on cluster • Results display in paragraph • Code and results can be shown or hidden
- 17. Create/open Note Note tools Paragraph tools User and note configuration Markdown Interpreter (%md) (editor hidden) Shell Interpreter (%sh) (editor shown)
- 18. MARKDOWN # headers %md hyperlink show/hide editor run paragraph run all paragraphs block quote
- 19. EXAMPLE • Crimes in Chicago Kaggle Dataset • Interesting opportunities for time series and prediction https://www.kaggle.com/currie32/crimes-in-chicago
- 20. DATA PIPELINE Acquire Kaggle Common Store Raw CSV zip Clean ORC Clean Explore Analyze
- 21. OPTIMIZING DATA CLEANING • Keep a raw copy • Web sites go away, remove data, change links and interfaces • Store the clean data • Saves time each time you analyze • Use a standard format (Optimized Row Columnar(ORC), parquet, etc) • Query data with hive • Shared location if security and privacy requirements allow • Collaborate by sharing data with others
- 22. ACQUIRE DATASET Acquire Kaggle Common Store Raw CSV zip Clean ORC Clean Explore Analyze
- 25. • %sh interpreter • Bash shell • Show intermediate results for debug
- 26. CLEAN DATASET Acquire Kaggle Common Store Raw CSV zip Clean ORC Clean Explore Analyze
- 29. SPARK IS FAST BUT LAZY • Transformations • Specify which data to read • Modify data • Actions • Show data • Write data
- 30. Header and Case data on Same CSV line
- 33. Apply numeric types On clean data Add some columns to make aggregations easier
- 34. Table for SQL Save clean data as ORC
- 35. EXPLORE DATASET Acquire Kaggle Common Store Raw CSV zip Clean ORC Clean Explore Analyze
- 36. Read clean data and create table
- 37. Specify query Select visualization Configure visualization X Y
- 45. OTHER INTERPRETERS • Matplotlib with pyspark • Angular – maps
- 47. ANALYZE DATASET Acquire Kaggle Common Store Raw CSV zip Clean ORC Clean Explore Analyze
- 48. CREATE DATA TO FIT POISSON
- 51. Evaluate Model
- 52. MODEL PIPELINES • https://spark.apache.org/docs/2.1.0/ml-pipeline.html TRANFORME RTransformers Estimator Pipeline Model Training data Test data Predictio ns
- 53. PYTHON – LOOKS LIKE SCALA
- 54. R
- 55. TIPS AND TRICKS • Use val for variables used across paragraphs • Vars can yield unpredictable results when run out of order • Break up big notebooks • Store intermediate results • Avoid reloading and recalculating the same values • Verify your notebook by running all paragraphs
- 56. SHARING NOTEBOOKS • Share link to notebook or paragraph • Readers access your Zeppelin server • Use logins and permissions • Export to JSON and save to shared file • Readers get JSON from shared file (github, cloud, etc) • Import to their Zeppelin server • Sync your to Zeppelin Hub (https://www.zeppelinhub.com) • Share Zeppelin Hub link with readers • Free version for small teams
- 57. REUSING NOTEBOOKS • Clone notebook • Copy code from notebook • Build libraries for use in notebooks
- 58. VERSIONING NOTEBOOKS • Track changes to notebook code or text • Go back to a previously known good version • Compare versions to see differences
- 59. CONFIGURE ZEPPELIN VERSION CONTROL 1 2 3 4 5Set storage to GitNotebookRepo in Zeppelin-env.xml Restart Zeppelin server
- 60. SAVE NOTE VERSION Enter version comment and click Commit
- 61. VIEWING VERSION HISTORY AND PREVIOUS VERSIONS Pull down the list of versions Select a version Zeppelin shows content for that version Head goes to the latest version
- 62. GIT REPO ON ZEPPELIN SERVER Zeppelin creates git repo In notebook directory
- 63. QUESTIONS AND THANK YOU!
- 64. REFERENCES
- 65. REPRODUCIBLE RESEARCH • Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten Simple Rules for Reproducible Computational Research. PLoS Comput Biol 9(10): e1003285. doi:10.1371/journal.pcbi.1003285 • http://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.p cbi.1003285&type=printable
- 66. ZEPPELIN AND SPARK • Spark • https://dzone.com/articles/try-the-latest-innovations-in-apache- spark-and-apa • https://hortonworks.com/hadoop-tutorial/learning-spark-zeppelin/ • https://spark.apache.org/docs/2.1.0/ml-pipeline.html • Example Notebooks • https://github.com/hortonworks-gallery/zeppelin-notebooks
- 67. ZEPPELIN INTERPRETERS • Markdown syntax • http://daringfireball.net/projects/markdown/syntax
- 68. EXAMPLE • Chicago Crimes Data Set • https://www.kaggle.com/currie32/crimes-in-Chicago • Example notebooks • https://github.com/carolynduby/ODSC2017