2019 swan-cs3
Download as PPTX, PDF0 likes295 views
The document outlines the SWAN analysis ecosystem, a cloud-based platform that supports various scientific analysis tools and simplifies the sharing of results. Key features include integration with CERN resources, a user-friendly interface, and collaborative capabilities through project sharing. Future developments aim to enhance user experience with new technologies and wider educational outreach.
2019 swan-cs3
- 1. SWAN and its analysis ecosystem D. Castro, J. Moscicki, M. Lamanna, E. Bocchi, E. Tejedor, D. Piparo, P. Mato, P. Kothuri Jan 29th, 2019 CS3 2019 - Cloud Storage Synchronization and Sharing Services https://cern.ch/swan
- 3. SWAN in a Nutshell ﹥Analysis only with a web browser No local installation needed Based on Jupyter Notebooks Calculations, input data and results “in the Cloud” ﹥Support for multiple analysis ecosystems ROOT, Python, R, Octave… ﹥Easy sharing of scientific results: plots, data, code ﹥Integration with CERN resources Sofware, storage, mass processing power 3
- 5. Storage ﹥Uses EOS disk storage system All experiment data potentially available ﹥CERNBox is SWAN's home directory Storage for your notebooks and data ﹥Sync&Share Files synced across devices and the Cloud Collaborative analysis 5
- 6. Software ﹥Software distributed through CVMFS ”LCG Releases” - pack a series of compatible packages Reduced Docker Images size Lazy fetching of software ﹥Possibility to install libraries in user cloud storage Good way to use custom/not mainstream packages Configurable environment 6 LCG Release CERN Software User Software Jupyter modules
- 7. Previously on last CS3 conference… 7
- 10. Sharing made easy ﹥Sharing from inside SWAN interface Integration with CERNBox ﹥Users can share “Projects” Special kind of folder that contains notebooks and other files, like input data Self contained 10
- 11. The Share tab ﹥Users can list which projects... they have shared others have shared with them ﹥Projects can be cloned to the receiver's CERNBox The receiver will work on his own copy ﹥Concurrent editing not supported by Jupyter Safer to clone 11
- 12. Spark Cluster Integration with Spark ﹥Connection to CERN Spark Clusters ﹥Same environment across platforms User data - EOS Software - CVMFS ﹥Graphical Jupyter extensions developed Spark Connector Spark Monitor Spark Master Spark Worker Python task Python task Python task User Notebook 12
- 14. The result 14
- 15. Stats ﹥~200 user sessions a day on average Users doubled last year with new SWAN interface ﹥~1300 unique users in 6 months ﹥Spark cluster connection: 15 – 20 % of users SWAN as entry point for accessing computational resources Used for monitoring LHC accelerator hardware devices (NXCals) 15 Courses
- 17. Inspecting a Project ﹥Users can inspect shared project contents Browsing of the files Static rendering of notebooks ﹥Useful to decide whether to accept or not the shared project 17
- 19. 1919Worldwide LHC Computing Grid (WLCG)
- 20. Connecting More Resources ﹥Ongoing effort: submit batch jobs from the notebook Monitoring display Jobs tab 20
- 22. Science Box: SWAN on Premises ﹥Packaged deployment of SWAN Includes all SWAN components: CERNBox/EOS, CVMFS, JupyterHub ﹥Deployable through Kubernetes or docker-compose ﹥Some successful community installations AARNet PSNC Open Telekom Cloud (Helix Nebula) 22
- 23. Science Box: SWAN on Premises ﹥UP2University European Project Bridge the gap between secondary schools, higher education and the research domain Partner universities (OU, UROMA, NTUA, …), pilot schools http://up2university.eu ﹥SWAN used by students to learn physics and other sciences Let them use the very same tools & services used by scientists at CERN Pilot with University of Geneva (Physiscope) ﹥Establishing collaboration with Callysto project 23
- 24. Looking ahead 24
- 25. Future work/challenges ﹥Move to Jupyterlab Porting the current extensions Concurrent editing ﹥New architecture Based on Kubernetes ﹥Exploitation of GPUs HEP is looking to ML Speed up computation of GPU-ready libraries (e.g. TensorFlow) 25
- 26. Where to find us 26
- 27. Where to find us ﹥Contacts swan-talk@cern.ch http://cern.ch/swan ﹥Repository https://github.com/swan-cern/ ﹥Science Box https://cern.ch/sciencebox 27
- 28. Conclusion 28
- 29. Conclusion ﹥Changes introduced since last year improved user experience Which translated on more users using the service ﹥SWAN became a fundamental Interface for Mass Processing Resources (Spark) Not only for Physics analysis but also for monitoring the LHC hardware ﹥The new Jupyterlab interface will bring new possibilities for collaborative analysis With the introduction of concurrent editing of notebooks Which can help reach more users ﹥Successfully deployed outside CERN premises Including on education related projects 29
- 30. SWAN and its analysis ecosystem Thank you Diogo Castro diogo.castro@cern.ch 30