JupyterCon 2017 - Collaboration and automated operation as literate computing for reproducible infrastructure
2 likes1,312 views
Yoshi Nobu Masatani from the National Institute of Informatics discusses the operations of the NII cloud operation team, focusing on enhancing reproducibility and collaboration through Jupyter Notebooks and automated processes. The document highlights the importance of literate computing for infrastructure reproducibility, emphasizing collaboration between engineers and users while maintaining traceability, reusability, and reproducibility. Key tools and methodologies are presented to support effective knowledge sharing and operational consistency.
![yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
Cell’s meme ID at `lc_cell_meme` of cell metadata.
{
"lc_cell_meme": {
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008", ## The meme id for this cell
"previous": "8f5c5cea-71cc-11e7-9abe-02420aff0008", ## The context of this cell as reference to the
"next": "f2125b84-4669-11e7-958b-02420aff0006", ## previous and the next cells
"history": [ ## The history of this notebook's context
{
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008",
"previous": "8f5c5ee8-71cc-11e7-9abe-02420aff0008",
"next": "8f5c60aa-71cc-11e7-9abe-02420aff0008"
},
{
"current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008",
"previous": "f2125b84-4669-11e7-958b-02420aff0006",
"next": "8f5c5ee8-71cc-11e7-9abe-02420aff0008"
}
]
},
...
}](https://image.slidesharecdn.com/jupytercon-masatani-170825211000/85/JupyterCon-2017-Collaboration-and-automated-operation-as-literate-computing-for-reproducible-infrastructure-31-320.jpg)
![yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
Notebook's meme ID at `lc_notebook_meme` of notebook metadata.
{
"lc_notebook_meme": {
"current": "1904d564-71c6-11e7-8369-0242ac110002", ## The meme id for this notebook
"lc_server_signature": {
"current": {
"server_url": "https://xxxxx.nii.ac.jp/user/xxxx/", ## The current notebook server information is also stored
"notebook_path": "/",
"notebook_dir": "/notebooks",
"signature_id": "034b406c-71c8-11e7-a8bf-02420aff0008"
},
"history": [
{ ## Travel history where this notebook has been executed
"notebook_dir": "/notebooks",
"notebook_path": "/",
"server_url": "http://localhost:8888/",
"signature_id": "dc3b0162-71bb-11e7-8369-0242ac110002"
}
]
},
...
}](https://image.slidesharecdn.com/jupytercon-masatani-170825211000/85/JupyterCon-2017-Collaboration-and-automated-operation-as-literate-computing-for-reproducible-infrastructure-32-320.jpg)
![yoshi NOBU masatani, National Institute of Informatics
https://github.com/NII-cloud-operation/Jupyter-LC_nblineage
The list of meme IDs of cells in this notebook is also recorded
{
"lc_notebook_meme": {
"current": "5fb59a5c-71d0-11e7-9acc-0242ac110002", ## newly assigned notebook's meme id
"lc_server_signature": {
"current": {
...
...
},
"root_cells": [
"5fb59caa-71d0-11e7-9acc-0242ac110002", ## The list of cells' meme IDs in this notebook
"5fb59dfe-71d0-11e7-9acc-0242ac110002", # In this case there are 6 cells in this notebook
"5fb59f2a-71d0-11e7-9acc-0242ac110002",
"5fb5a042-71d0-11e7-9acc-0242ac110002",
"5fb5a150-71d0-11e7-9acc-0242ac110002",
"5fb5a25e-71d0-11e7-9acc-0242ac110002"
],
"history": [
"1904d564-71c6-11e7-8369-0242ac110002" ## old meme id of this notebook that was <source.ipynb>
]
},
...
}](https://image.slidesharecdn.com/jupytercon-masatani-170825211000/85/JupyterCon-2017-Collaboration-and-automated-operation-as-literate-computing-for-reproducible-infrastructure-33-320.jpg)
JupyterCon 2017 - Collaboration and automated operation as literate computing for reproducible infrastructure
- 2. yoshi NOBU masatani, National Institute of Informatics NII Cloud Operation • NII Cloud Operation is a team supporting researchers and teachers in our institute • Struggling to adequately support; maintaining private OpenStack, setting up software stacks, consulting their configuration and optimization, and managing almost everything • We have been applying the Jupyter Notebook to operation engineering for years https://github.com/NII-cloud-operation
- 3. yoshi NOBU masatani, National Institute of Informatics • our domain and motivation • extension to Jupyter • what have learned and where we are going to..
- 4. yoshi NOBU masatani, National Institute of Informatics Literate Computing for Reproducible Infrastructure • Automated Operation: assure traceability and reproducibility utilizing computational narratives, i.e. Jupyter Notebook • Collaboration: share reproducible experience participating both tech and non-tech alike regarding infrastructure design, development and operation
- 5. yoshi NOBU masatani, National Institute of Informatics Facing Practical Problems.. • Record daily operations for assuring ops quality • Refactor proven but scattered procedures into reusable forms • Prepare knowledge for apprentice engineers and serve educational materials for users At all situations above computing and engineering experience should be consistent and seamless... Beyond SILOs!! • for exchanging and accumulating knowledge • similar descriptive methods, i.e. language and representation, should be applicable • efficient granularity and representation of schema should be managed • would not depend upon subjects/objects matters, phases of processes.. Traceability Reusability Reproducibility
- 6. yoshi NOBU masatani, National Institute of Informatics Reproducible Experience Reproducible Operation Procedure Make “Prerequisite” reproducible too What we call as “Bindings” to the world. • Moving targets, i.e. systems’ system • Always involve human in a evolutional loop Literate Computing Skill 文芸 Art How we maintain Notebooks Reproducible Infrastructure Share Knowledge Share Reproducible Experience Jupyter Notebook ~ Reproducible Document
- 7. yoshi NOBU masatani, National Institute of Informatics Literate Computing for Reproducible Infrastructure Cloud < Live Codes > < Outputs > refactor operations run notebooks as live codes describe reproducible operations as notebooks keep outputs for tractability and reproducibility communicate via notebookcommunicate via notebook share and elaborate narrative stories share and elaborate narrative stories self administration reproducible workflows accelerated experiments Ansible Playbook & Jupyter Notebook Reference Patterns Expert EngineerApprentice Engineer Expert User Participant User
- 8. yoshi NOBU masatani, National Institute of Informatics Infrastructure Operation
- 9. yoshi NOBU masatani, National Institute of Informatics Daily Operation through Jupyter Evidences for traceable operations
- 10. yoshi NOBU masatani, National Institute of Informatics Reusable Notebooks for Reproducible Infrastructure 25 Notebooks & 55 Playbooks • Deployment • Maintenance/Operation • Test/Verify https://github.com/NII-cloud-operation/Literate-computing-Hadoop
- 11. yoshi NOBU masatani, National Institute of Informatics Our Deployment Expert Engineer @Operation Team Production @Private Cloud Playground @Public Cloud Participant User Apprentice Engineer @Operation Team JupyterHub Expert User Jupyter @Management Server Managed by Operators Self Admin by Users Acquire Skills from Experiences on Sandbox KVM … Jupyter @Hypervisor
- 12. yoshi NOBU masatani, National Institute of Informatics Share Reproducible Experiences Transfer skill and knowledge through shared reproducible experiences among engineers, users, tech and non-tech. < Live Codes > < Outputs > Engineers Expert Engineer share experiences Apprentice Engineer Expert User Participant User User elaborate notebookstest deployment of applications production deployment for users share experiences for administration share experience acquire skill and knowledge through shared experiences self administration
- 13. yoshi NOBU masatani, National Institute of Informatics Enhancement for Robust and Reliable Operation The goals for Literate Computing tools are: • Secure outputs; summarize massive output lines on GUI and save all original output lines into an individual file • Prevent miss-operation; once a cell has been executed, it “freezes” against unintended execution. Also you can “lock” cells for unintended modification. • Add a perspective; when markdown’s hierarchy is collapsed embedded cells underneath are represented as “bricks” and also run through altogether. Jupyter Notebook server which bundles a set of tools for Literate Computing for Reproducible Infrastructure. This bundle shows what we have enhanced Jupyter. docker run -it --rm -p 8888:8888 niicloudoperation/notebook
- 14. yoshi NOBU masatani, National Institute of Informatics About NII Extensions https://github.com/NII-cloud-operation
- 15. yoshi NOBU masatani, National Institute of Informatics Secure Outputs – LC_wrapper kernel • Summarize massive output lines • At each cell’s execution all original output lines are saved into an individual file with a time stamp. • You can review whole output and compare with previous results from different executions. https://github.com/NII-cloud-operation/Jupyter-LC_wrapper
- 16. yoshi NOBU masatani, National Institute of Informatics Enhanced Collapsible Headings https://github.com/NII-cloud-operation/Jupyter-LC_run_through/
- 17. yoshi NOBU masatani, National Institute of Informatics “Run through” Control
- 18. yoshi NOBU masatani, National Institute of Informatics Notebook for Diagnostic & Variation
- 19. yoshi NOBU masatani, National Institute of Informatics Visualize Relationship and Dependency https://github.com/NII-cloud-operation/Literate-computing-Elasticsearch/blob/master/99_Summarizing%20notebooks.ipynb http://blockdiag.com/en/
- 20. yoshi NOBU masatani, National Institute of Informatics Visualize Relationship and Dependency
- 21. yoshi NOBU masatani, National Institute of Informatics Visualize Relationship and Dependency
- 22. yoshi NOBU masatani, National Institute of Informatics Exploring Best Way..
- 23. yoshi NOBU masatani, National Institute of Informatics https: //github.com/NII-cloud-operation/Jupyter-LC_docker << Document $ docker pull niicloudoperation/notebook $ docker run -it --rm -p 8888:8888 niicloudoperation/notebook demo
- 24. yoshi NOBU masatani, National Institute of Informatics Automation Should Be Like Iron Man, Not Ultron “The automation we created abided by the complementarity principle. It was a partnership between human and machine. It did not limit our ability to learn and grow. The control over the system was shared between the automation and the humans involved. In other words, rather than creating a system that took over the cluster and ran it, we created one that partnered with humans to take care of most of the work. It did its job autonomously, but we did not step on each other's toes.” Thomas A. Limoncelli. 2015. Automation Should Be Like Iron Man, Not Ultron. Queue 13, 8, pages 50 (September 2015), 10 pages.
- 25. yoshi NOBU masatani, National Institute of Informatics Collaboration and Automated Operation “Manual” Operations “Autonomous” Operations No Ops “Automated” Operations DevOps + Programming “Leftover Principle” “Complementarity Principle” Automation within a Silo and in Blackboxes Literate Computing for Reproducible Infrastructure + Computational Narratives • Collaboration among human sharing reproducible experiences including both “what went wrong” and “what went right” • Collaboration with Automation keeping potential to respond, monitor, learn and anticipate
- 26. yoshi NOBU masatani, National Institute of Informatics Our Tools as Parts of Ourselves “Automation weakens the bond between tool and user not because computer-controlled systems are complex but because they ask so little of us. They hide their workings in secret code. They resist any involvement of the operator beyond the bare minimum. They discourage the development of skillfulness in their use. Automation ends up having an anesthetizing effect. We no longer feel our tools as parts of ourselves. ” InCarr, Nicholas. The Glass Cage: How Our Computers Are Changing Us (p. 223). W. W. Norton & Company. Kindle Edition.
- 27. yoshi NOBU masatani, National Institute of Informatics Hardiman, General Electric in 1965 https://www.engadget.com/2014/01/26/ge-man-amplifying-robots/ Automation 自働化 ≠ Automated Operation 機械化 “Our tools as parts of ourselves”
- 28. yoshi NOBU masatani, National Institute of Informatics Hadoop – Ready! on CentOS6 / Hadoop Swimlanes – Ready! on Tez / Hivemall – Ready! on Hive / … Start / Stop the services / Hadoop – Decommission DataNode / … Operational Policy Settings; Security.../ Hadoop Prerequisites – Ready! on CentOS6 KVM – Ready! on CentOS / KVM – Set! CentOS6 / KVM – Go! VM / … CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 CentOS6 VM / Baremetal VM / Baremetal VM / Baremetal VM / Baremetal VM / Baremetal VM / Baremetal VM / Baremetal VM / Baremetal ZooKe eper ZooKe eper ZooKe eper NameN ode NameN ode DataNo de DataNo de DataNo de DataNo de Resour ce Manag er Resour ce Manag er Node Manag er Node Manag er Node Manag er Node Manag er Spark Tez- tools Hive Hivem all Test Dataset Test Job 3. Hadoop Deployment Notebook 4. Hadoop Operation Notebook 2. Hadoop Prerequisites Notebook Simple YARN for Test / Simple Spark script for Test / … 5. Hadoop Post Verification Notebook 1. Hadoop Cluster Design NotebookSet! Inventory YARN HDFS Elaborated Notebook’s Structure
- 29. yoshi NOBU masatani, National Institute of Informatics Elaborated Control Flows
- 30. yoshi NOBU masatani, National Institute of Informatics nblineage – Understand Life Cycle of Notebooks How people share, revise, and evolve notebooks in a team. A hypothesis: - there should be a seed notebook and it would be reused and revised several situations. - Subsequently, a useful seed would generate a collection of diverged notebooks, - Then the collection would be refactored.. nblineage has been developed in order to study how usable notebooks are evolved and distributed. This is on-going practice..
- 31. yoshi NOBU masatani, National Institute of Informatics https://github.com/NII-cloud-operation/Jupyter-LC_nblineage Cell’s meme ID at `lc_cell_meme` of cell metadata. { "lc_cell_meme": { "current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008", ## The meme id for this cell "previous": "8f5c5cea-71cc-11e7-9abe-02420aff0008", ## The context of this cell as reference to the "next": "f2125b84-4669-11e7-958b-02420aff0006", ## previous and the next cells "history": [ ## The history of this notebook's context { "current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008", "previous": "8f5c5ee8-71cc-11e7-9abe-02420aff0008", "next": "8f5c60aa-71cc-11e7-9abe-02420aff0008" }, { "current": "8f5c5fe2-71cc-11e7-9abe-02420aff0008", "previous": "f2125b84-4669-11e7-958b-02420aff0006", "next": "8f5c5ee8-71cc-11e7-9abe-02420aff0008" } ] }, ... }
- 32. yoshi NOBU masatani, National Institute of Informatics https://github.com/NII-cloud-operation/Jupyter-LC_nblineage Notebook's meme ID at `lc_notebook_meme` of notebook metadata. { "lc_notebook_meme": { "current": "1904d564-71c6-11e7-8369-0242ac110002", ## The meme id for this notebook "lc_server_signature": { "current": { "server_url": "https://xxxxx.nii.ac.jp/user/xxxx/", ## The current notebook server information is also stored "notebook_path": "/", "notebook_dir": "/notebooks", "signature_id": "034b406c-71c8-11e7-a8bf-02420aff0008" }, "history": [ { ## Travel history where this notebook has been executed "notebook_dir": "/notebooks", "notebook_path": "/", "server_url": "http://localhost:8888/", "signature_id": "dc3b0162-71bb-11e7-8369-0242ac110002" } ] }, ... }
- 33. yoshi NOBU masatani, National Institute of Informatics https://github.com/NII-cloud-operation/Jupyter-LC_nblineage The list of meme IDs of cells in this notebook is also recorded { "lc_notebook_meme": { "current": "5fb59a5c-71d0-11e7-9acc-0242ac110002", ## newly assigned notebook's meme id "lc_server_signature": { "current": { ... ... }, "root_cells": [ "5fb59caa-71d0-11e7-9acc-0242ac110002", ## The list of cells' meme IDs in this notebook "5fb59dfe-71d0-11e7-9acc-0242ac110002", # In this case there are 6 cells in this notebook "5fb59f2a-71d0-11e7-9acc-0242ac110002", "5fb5a042-71d0-11e7-9acc-0242ac110002", "5fb5a150-71d0-11e7-9acc-0242ac110002", "5fb5a25e-71d0-11e7-9acc-0242ac110002" ], "history": [ "1904d564-71c6-11e7-8369-0242ac110002" ## old meme id of this notebook that was <source.ipynb> ] }, ... }
- 34. yoshi NOBU masatani, National Institute of Informatics Lineage Network Notebook degree of common memes
- 35. yoshi NOBU masatani, National Institute of Informatics Lineage Network
- 36. yoshi NOBU masatani, National Institute of Informatics Lineage Network Collect reusable materials as a baseline CentOS 6 7 Import stacks Deploy a seed Production Side work for production Another concurrent development
- 37. yoshi NOBU masatani, National Institute of Informatics Retrospective Communication
- 38. yoshi NOBU masatani, National Institute of Informatics Thank you https://github.com/NII-cloud-operation