Apache Zeppelin and Helium @ApacheCon 2017 may, FL
2 likes602 views
The document discusses Apache Zeppelin, a web-based notebook for interactive data analytics that supports multiple programming languages and features such as real-time collaboration and built-in visualizations. It highlights challenges around adding new visualizations and interpreters and proposes solutions through pluggable visualizations and online registries for helium packages. Future plans include releasing an online registry and improved interpreter functionality in version updates.
Apache Zeppelin and Helium @ApacheCon 2017 may, FL
- 1. Helium makes Zeppelin Fly Moon soo Lee (moon@zepl.com) Hoon Park (1ambda@zepl.com) Ahyoung Ryu (ahyoungryu@zepl.com) @ZEPL
- 2. Who we are?
- 3. What is Apache Zeppelin? A web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.
- 4. When do you need Zeppelin? Engineers Data Scientists Business user Visualizations Access control Report When multiple tools and different peoples are involved in your data pipeline JDBC
- 5. Notebook Zeppelin allows use multiple interpreter (language backend) at the same time ● Real-time collaboration ● Authentication ● Notebook ACL ● Interpreter ACL ● Fine grained interpreter session - notebook/user mapping ● Built-in scheduler ● Pluggable notebook storage ● Pluggable interpreter Zeppelin supports multi-user environment
- 6. Built-in Visualization 6 Basic visualizations are built-in, (in addition to matplotlib, ggplot integration) (Table, Bar chart, Pie chart, Area chart, Scatter chart, Line chart)
- 8. Why isn’t it easy to add new visualizations? - Dependent on Zeppelin release cycle - Restricted License (e.g. commercial chart) Problems: Built-in Visualizations “What if I want to display things differently?”
- 9. Let’s add pluggable visualizations - external add-ons - can update frequently - highly customizable - shared via online registry Solution? Problems: Built-in Visualizations Why isn’t it easy to add new visualizations? - Restricted License (e.g. highchart) - Dependent on Zeppelin release cycle liumHe 2 4.0026
- 12. “What about extending Interpreters?” “Can we leverage Helium framework for interpreters?”
- 13. Backgrounds: Interpreter Execute paragraph (code) and return output - separated JVM process - %spark, %jdbc, %python, ...
- 15. It is not easy to add a new interpreter - interpreters written in Java - sometimes need to handle HTML dynamically Can’t be combined with other interpreters - e.g. Spark + Markdown - Interpreter != Display System Problems: Backend Interpreter “Can we write interpreters easily?”
- 16. Backgrounds: Display System customize interpreted outputs in frontend - can be combined with interpreters - %html, %table, %angular, ...
- 17. Problems: Backend Interpreter “Frontend interpreter can be a display system”
- 18. Easy to create and handle HTML - written in JS - can utilize many existing JS libraries (flowchart, sigmajs, vega, papaparse, ...) Can be a display system like %html, %table - e.g. Spark Interpreter + Markdown Display - allows to customize output %myGraph SPELL: Frontend Interpreter
- 21. lium Online RegistryHe 2 4.0026 Ahyoung Ryu @ ZEPL linkedin.com/in/AhyoungRyu
- 22. How can we share VISUALIZATION or SPELL packages?
- 26. Online Registry for Helium packages?
- 27. THEN WE SHOULD CONSIDER 1. Who/ How can we build up the infra and operate the service? - Need to set up the authentication system - Need user/ package DB - Versioning/ building / packaging - … 2. External library Licenses
- 28. Too complicated.. Is there any other SIMPLE way to solve this?
- 29. A SOLUTION ALREADY EXISTED
- 30. A SOLUTION ALREADY EXISTED Helium VISUALIZATION & SPELL packages are package
- 31. A SOLUTION ALREADY EXISTED The package information can be saved in registry! Helium VISUALIZATION & SPELL packages are package
- 32. Then, is there any way to fetch ONLY Helium package information from registry?
- 35. HOW? Publish it to Registry Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 36. HOW? Publish it to Registry Does the package has zeppelin-vis or zeppelin-spell as its dependency? Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 37. HOW? Publish it to Registry If so, take only necessary metadata: - name - description - version - license - ... Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 38. HOW? Publish it to Registry Integrate whole data and create helium.json Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 39. HOW? Publish it to Registry Integrate whole data and create helium.json Save the file in Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 40. HOW? Publish it to Registry Save the file in Trigger Lambda function every 1 hour using Integrate whole data and create helium.json Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 41. HOW? Publish it to Registry Read helium.json Integrate whole data and create helium.json Trigger Lambda function every 1 hour using Save the file in Filter Helium pkg & fetch only necessary metadata Create a Helium package
- 44. JIRA ISSUES - ZEPPELIN-1973 : List all available Helium packages in Zeppelin website - ZEPPELIN-2004 : List helium packages in Zeppelin GUI by reading file
- 45. WHEN CAN I USE THIS? - Not included in Zeppelin latest version 0.7.X - Will be available in Zeppelin 0.8.0 - Release plan ?
- 46. extends Zeppelin eco-system Interpreters liumHe 2 4.0026 Visualizations Spell Map Heatmap Range Bubble Spline Sigma D3 Markdown Translator Flowchart Spark Python JDBC Groovy Geode FlinkCassandra Kylin Users / Developers 2833 3rd parties
- 47. Useful service Zeppelin notebook online viewer https://www.zeppelinhub.com/viewer Notebook sharing and collaboration https://www.zeppelinhub.com
- 48. Future Roadmap 0.7.2 - Maintenance release. 0.8.0 - Helium online registry. - Interpreter Cluster mode. 1.0 - Finest, the most stable release 2Q /2017 3Q /2017 3-4Q /2017 Hopefully
- 49. Thanks Q&A