Introduction to Globus - XSEDE14 Tutorial
1 like572 views
Globus is a SaaS solution designed for the secure and efficient transfer and sharing of large research data, facilitating easy access to storage systems without requiring cloud storage. It provides features like automated file transfers, user permissions, and an extensive toolkit for data management, which is increasingly adopted in research institutions. The platform also aims to enhance data publication and discovery through upcoming services focused on managing and curating research datasets.
Introduction to Globus - XSEDE14 Tutorial
- 1. Globus Introduction Raj Kettimuthu Steve Tuecke Vas Vasiliadis
- 2. “I need a good place to store / backup / archive my (big) research data, at a reasonable price.” Public Cloud ArchiveMass StoreCampus Store
- 3. “I need to easily, quickly, & reliably move or mirror portions of my data to other places.” Research Compu.ng HPC Cluster Lab Server Campus Home Filesystem Desktop Worksta.on Personal Laptop XSEDE Resource Public Cloud
- 4. “I need to easily and securely share my data with my colleagues at other institutions.”
- 5. “I need to get data from a scientific instrument to my analysis server.” Next Gen Sequencer Light Sheet Microscope MRI Advanced Light Source
- 6. Challenge: Manage research data as easily as… …our pictures …home entertainment …our e-‐mail
- 7. What is Globus? Big data transfer, and sharing… … delivered via SaaS … … that is simple, secure, and fast… … directly from your own storage systems
- 8. Reliable, secure, high-performance file transfer & synchronization • “Fire-and-forget” transfers • Automatic fault recovery • Seamless security integration • Powerful GUI and APIs Data Source Data Destination User initiates transfer request 1 Globus moves and syncs files 2 Globus notifies user 3
- 9. Simple, secure sharing off existing storage systems Data Source User A selects file(s) to share, selects user or group, and sets permissions 1 Globus tracks shared files; no need to move files to cloud storage! 2 User B logs in to Globus and accesses shared file 3 • Easily share large data with any user or group • No cloud storage required
- 10. Globus is SaaS • Web, command line, and REST interfaces • Reduced IT operational costs • New features automatically available • Consolidated support & troubleshooting • Easy to add your laptop, server, cluster, supercomputer, etc. with Globus Connect
- 11. 8,000 active endpoints (in the past year)
- 13. Globus increasingly used to build campus-wide data services Source: University of Nebraska Holland Computing Center Enable campus compu4ng facili4es to be7er u4lize high performance network infrastructure
- 14. 10GE 10GE 10GE 10GE 10G Border Router WAN Science DMZ Switch/Router Enterprise Border Router/Firewall Site / Campus LAN Per-service security policy control points Clean, High-bandwidth WAN path Site / Campus access to Science DMZ resources perfSONAR perfSONAR perfSONAR High performance Data Transfer Node with high-speed storage Typical deployment Science DMZ + Globus Details at: fasterdata.es.net
- 15. Demonstration
- 16. 1. Go to: globus.org/signup 2. Create your Globus account 3. Validate e-mail address 4. Optional: Login with your campus/InCommon identity Exercise 1: Account Signup
- 17. 1. Install Globus Connect Personal 2. Move file(s) from esnet#anl-diskpt1 to your laptop 3. Sign up for a free Globus Plus trial 4. Create a shared endpoint on your laptop 5. Grant your neighbor permissions on your shared endpoint 6. Access your neighbor’s shared endpoint 7. Optional: Create group, and grant share access Exercise 2: Transfer, Sharing, Group Management
- 18. Our challenge: Sustainability We are a non-profit, delivering a production-grade service to the non-profit research community
- 19. Globus Provider Subscriptions • Managed Endpoints – Priority support – Management console – Usage reports – Mass Storage System optimization – Host shared endpoints – Integration support • Branded Web Site • Alternate Identity Provider (InCommon is standard) globus.org/provider-plans
- 20. NET+ Globus • Internet2 members get discounted Globus Provider subscriptions • Completing “Service Validation” phase – Sponsors: Cornell, U.Michigan, Yale, U.Missouri, and U.Chicago • Available to “Early Adopters” soon
- 21. Globus Platform-as-a-Service Identity, Group, Profile Management Services … Sharing Service Transfer Service Globus Toolkit GlobusAPIs GlobusConnect
- 22. globus genomics Flexible, scalable, affordable genomics analysis for all biologists
- 23. + Data management PaaS Next-gen sequence analysis SaaS + Scalable IaaS
- 24. Globus is moving beyond transfer and sharing to data publication and discovery
- 25. Globus Data Publication (coming soon) • SaaS for publishing large research data • Bring your own storage • Extensible metadata • Publication and curation workflows • Public and restricted collections • Rich discovery model
- 27. Search Browse Access …across collections, endpoints …and facilitates rich discovery
- 28. Metadata Access Control License Storage Curation Workflow Policies Collection Globus’ view of data publishing Metadata DataMetadata Data Metadata Data Dataset Dataset Dataset Community
- 29. Argonne Storage System Univ. of Chicago Argonne IIT UIUC Exemplar Use Case 3. Assemble Dataset (Transfer Data) Argonne Curator 2. Describe Submission Scientist Shared Endpoint 4. Curate Dataset 1. Publish Data 6. Download 5. Search
- 31. 1. Optional: Generate SSH key 2. Go to: globus.org/account/ ManageIdentities 3. Add your SSH key to your Globus identity 4. SSH to cli.globusonline.org 5. Check on the status of your earlier transfer(s) 6. Transfer a file using the scp command Exercise 3: Globus CLI
- 32. Thank you to our sponsors! U . S . D E P A R T M E N T O F ENERGY