Web Data Analysis at the Spallation Neutron Source
Download as PPTX, PDF0 likes40 views
This document summarizes a proposal for building a web-based data reduction platform for small angle neutron scattering. The key points are: 1) The platform would allow users to perform data reduction from a desktop, tablet or phone using responsive web design. It would support both expert and regular users with different permissions. 2) The requirements include spreadsheet-style input, script generation from the user input, parallel job execution, and preliminary analysis capabilities. 3) Technologies proposed include Django, Bootstrap, datatables, Handsontable, Highlight.js, Vega/VegaLite, and Plotly for plotting, along with responsive templates, Electron for desktop apps, Docker, and modern web standards.
Web Data Analysis at the Spallation Neutron Source
- 1. ORNL is managed by UT-Battelle for the US Department of Energy Web Data Analysis at the Spallation Neutron Source Ricardo M. Ferraz Leal Neutron Data Analysis and Visualization Group rhf@ornl.gov
- 2. 2 Motivation: why not? • SNS has several Web Applications: – People DO use them! • Clusters available to process data. • ICAT: – REST API provides experiment information • Data files path, metadata, IPTS • ICAT + LDAP: – Authentication + Permissions • No need to download Data Analysis Software • Developer: End of cross compilation! • People love Facebook!
- 3. Building a Web Reduction platform A Small Angle Neutron Scattering prototype…
- 4. 8 Requirements • Run from a desktop computer, tablet or phone! • Two user modes: – Expert (beamline scientists, advanced users) – Regular users • Data access permissions • Spreadsheet style input – Some scientists use openpyxl to read excel spreadsheets! • Script generation (and tweaking) from the user input • Parallel job execution • Preliminary analysis (e.g. simple fittings)
- 5. 9 • Responsive Web Design Template: – Google “responsive web design template dashboard” • About 565.000 results – AdminLTE: • https://almsaeedstudio.com/themes/AdminLTE/index2.html • https://github.com/almasaeed2010/AdminLTE “Shopping for free…”
- 7. 11 Bootstrap
- 10. 14 Highlight.js
- 11. 15 • Json visualization grammar – Vega + VegaLite: • https://vega.github.io/vega-lite/ • New Future standard for json plot format (?) • Python backend with HTML/JS rendering. – Altair: • https://github.com/altair-viz/altair • Uses VegaLite • New! – Bokeh: • http://bokeh.pydata.org/en/latest/ • Still in development… – Plotly Offline: • https://plot.ly/python/offline/ • Now free and stable Plotting: D3 & friends
- 13. 17 • Model Based Views: – Generic Views: List, Create, Update, Delete, etc.. – Model Based Forms • Admin Interface • Initial release: 21 July 2005; 11 years ago – Out of the box Django is a little outdated: • Server side template system • Every request / action loads a full page. • Django Channels (2016): – WebSockets and HTTP2 – Will ship (?) with Django 1.11 (2017) Django
- 14. 18
- 15. Web based Data Reduction: Putting the shopping cart together… Small Angle Neutron Scattering Data Reduction & Analysis
- 16. 20 • Configuration: – Advanced parameters • Masks, calibration, etc… – Can access files from any experiment (e.g mask file): • Server side file browser – A configuration can be assigned to other users • Reduction: – Uses a Configuration – Minimal user input • Uses as much metadata as possible • Form contents can be inferred from the metadata Configuration vs Reduction
- 17. 21
- 18. 22
- 19. 23
- 20. 24
- 21. 25
- 22. 26
- 23. 27
- 24. 28
- 26. 30
- 27. New Version: 2016 Web Development Standards
- 28. 32 • Goal: – To provide a UX similar to that of a desktop application. A single-page application (SPA) <!DOCTYPE html> <html lang="en"> <head> </head> <body> <div id="app"></div> <script src="/dist/build.js"></script> </body> </html> See tips: “I am a fast webpage, kneel before my greatness” https://varvy.com/pagespeed/wicked-fast.html
- 29. 33 • Effortless integration with Electron • Cross platform desktop apps with JavaScript, HTML, and CSS Cross platform desktop app http://electron.atom.io/apps/
- 30. 34 • JavaScript Framework: – Vue.js – Webpack, node.js & friends… • REST API – Django + REST Framework • Postgres >= 9.4 jsonb (Django >= 1.9 JSONField): – Allows storing form content in JSON format • NoSQL Functionality • Docker containers Tools…
- 32. 36
- 33. 37 Wanna play? Let me know: rhf@ornl.gov Acknowledgements: – Tanner Hobson – Shelly Ren – Peter Parker – Ray Gregory – NDAV Group – EQSANS, BioSANS, GPSANS teams
Editor's Notes
- #2: Web data analysis: Data analysis on the web The user doesn’t need to download any dedicated data analysis software! Finished the code that runs on your local machine, but doesn’t build on windows or osx
- #9: Went around and asked BL scientists and users what were their requirements Script generation : reduction software independent, facility independent
- #10: Github was a great contribution to the open source community People like to show off
- #17: Because we were already using Django… Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Fast Secure: Forgives some programmers mistakes,,,, Cross-Site Request Forgery (CSRF), cleans forms (SQL injection) Beauty of Django consists in the models: - Simple python classes that map to tables in the database. Objects are entries in that table.
- #26: Table populated by the users but will be automatically populated by ICAT with the new experiment planning and tagging.
- #33: SPA: - One Single HTTP request - Stile sheets + images in base 64 in the single JS file
- #35: Vue.js : - New kid on the block similar to Angular, React, Ember, Backbone Django does not provide HTML any longer. Now provides JSON!
- #36: Images are inlined in base64 to decrease the number of HTTP requests