Simplified Research Data Management with the Globus Platform
1 like113 views
The document provides an overview of the Globus platform's capabilities for research data management, highlighting its history, core functions, and features such as secure file sharing, automated data movement, and data publication. It emphasizes the platform's flexibility in accessing diverse storage systems and its role in supporting researcher needs for data stewardship and automation. Additionally, the document discusses themes of sustainability, security in data handling, and the platform's growing user base and statistics.
Simplified Research Data Management with the Globus Platform
- 1. Simplified Research Data Management with the Globus Platform CNI Membership Meeting, Fall 2018 - Project Update Vas Vasiliadis vas@uchicago.edu
- 2. Topics âą What is Globus? âą Globus from a researcherâs perspective âą Common use cases: research data automation âą Data publication with Globus âą Sustainability â itâs in our DNA
- 3. Research data management today (circa 2008) How do we... ...move, share, describe, discover, reproduce? Index? Facilitate data stewardship
- 4. Globus: A Brief History of Time âą Oct. 1998 â Globus Toolkit v1.0.0 âą Nov. 2010 â Globus Online initial release âą Nov. 2013 â Sustainability model launched âą Dec. 2016 â 50,000 registered users, 200PB+ moved âą Jan. 2018 â Globus Toolkit support EOL âą Jan. 2019 - 100th subscriber signed, >50% sustainable âą ??? â Globus becomes fully self-sustaining globus online
- 5. Globus⊠bridges data and people within and beyond organizational boundaries
- 6. 6 Research Computing HPC Desktop Workstations Mass Storage Instruments Personal Resources Public/Private Cloud National Resources Unified access to data across storage tiers
- 7. Public / private cloud stores External campus storage Project repositories, replication stores Public repositories Sharing with collaborators, community
- 8. Globus: Core functions Researcher initiates transfer request; or requested automatically by script, science gateway 1 Instrument Compute Facility Globus transfers files reliably, securely 2 Globus controls access to shared files on existing storage; no need to move files to cloud storage! 4 Curator reviews and approves; data set published on campus or other system 7 Researcher selects files to share, selects user or group, and sets access permissions 3 Collaborator logs in to Globus and accesses shared files; no local account required; download via Globus 5 Researcher assembles data set; describes it using metadata (Dublin core and domain- specific) 6 6 Peers, collaborators search and discover datasets; transfer and share using Globus 8 Publication Repository Personal Computer Transfer Share Publish Discover âą Use a Web browser âą Access any storage âą Use an existing identity
- 23. Globus for high assurance data management âą Restricted data handling: PHI, PII, CUI âą Security controls: NIST 800-53, 800-171 Low âą Business Associate Agreement (BAA) w/UChicago â University of Chicago has a BAA with Amazon
- 24. High Assurance features âą Additional authentication assurance â Per storage gateway policy on frequency of authentication with specific identity for access to data â Ensure that user authenticates with the specific identity that gives them access within session âą Application instance isolation â Authentication context is per application, per session âą Encryption of user data in transit and Globus data at rest âą Detailed audit log (on data transfer nodes)
- 28. Accessing Globus from your own storage
- 29. âŠclient software that makes a storage system accessible via Globus
- 30. Globus Connect Personal âą Installers do not require admin access âą Zero configuration; auto updating âą No firewall changes required; handles NATs
- 31. Globus Connect Server âą Installed and managed by sysadmin âą Default access for all local accounts 31 docs.globus.org/globus-connect-server-installation-guide/ Local system users Local Storage System (HPC cluster, NAS, âŠ) Globus Connect Server MyProxy CA GridFTP Server OAuth Server Data Transfer Node âą POSIX + connectors âą Native packaging Linux: DEB, RPM
- 32. IBM Spectrum Scale Current Planned Storage Connectors - globus.org/connectors ActiveScale
- 33. Use(r)-appropriate interfaces GET /endpoint/go%23ep1 PUT /endpoint/vas#my_endpt 200 OK X-Transfer-API-Version: 0.10 Content-Type: application/json ... Globus service Web CLI Rest API
- 34. Globus Command Line Interface âą Full-featured (web++) âą Uses Python SDK âą Open source github.com/globus/globus-cli docs.globus.org/cli
- 35. Globus is PaaS⊠âŠfor building science gateways, portals, and other web applications in support of research and education
- 36. Globus Auth (identity and access management) ⊠GlobusAPIs (Transfer,Search,Identifiers,âŠ) GlobusConnect Data Publication File Sharing File Transfer, Sync The Globus Platform Data Automation
- 37. Globus Auth âą Foundational Identity and Access Management (IAM) service âą Protects REST API communications âą Enables login for diverse app ecosystem, no new identity required âą Employs least privileges security model Auth User Authentication Secure service interactions Application identity and interactions
- 38. Globus helps automate data flows
- 39. Automated data movement 39 Scheduled backup My Drive/backups |__/20170930 |__/20170929 |__/20170928 |__âŠ. Recurring transfers with sync option Campus Lab
- 40. Streamlined data distribution My Drive/projectX |__/source |__/pipe0001 |__/pipe0002 |__âŠ. Secure sharing with research community Discover and access via data portalHPC resource, Campus storage, âŠ
- 41. Data distribution example: NCAR RDA
- 42. Reliable instrument data egress My Drive/FASTQ |__/cohort_0_0 |__/cohort_0_1 |__/cohort_0_2 |__âŠ. Stage data for downstream analysis NGS and high-res Imaging (APS, ALS, CryoEM, fMRI,âŠ)
- 43. Instrument data egress example âą Kasthuri Lab at Uchicago: brain aging and disease âą Construct connectome (map neuron connections)
- 44. JLSEUChicago ALCFAPS Publication7 Kasthuri Lab neuroanatomy reconstruction pipeline Imaging1 Lab Server 1 Acquisition2 Lab Server 2 Pre-processing3 Preview/Center4 Reconstruction6Visualization8 User validation5 Science!9
- 45. Data Management Plan enablement My Drive/datasets |__/afdb4523 |__/235fabcc |__/cd23a421 |__âŠ. Dataset assembly, description, curation http://hdl.handle.net/11466/OMN5BFB Access via persistent identifier Diverse storage systems
- 46. Globus Data Publication V1 âą Cloud-based web app âą BYO storage âą User-managed collections âą Select pre-defined schema âą Handle, DOI persistent identifiers âą >2000 users, >600 datasets publish.globus.org
- 47. Many variations of data publication⊠Citable Data ⹠Standard metadata ⹠Persistent identifiers ⹠Durable storage ⹠Many domains ⹠Custom metadata ⹠Locally managed storage Institutional Data ⹠Agreed schema ⹠Larger datasets ⹠Fine grained metadata Community Data
- 48. âŠIncluding active data management Active Research Data âą Less standard and evolving schema âą Data organized independent of storage âą Support active collaboration âą Location agnostic identifiers
- 49. Publication v2 Platform âą Decompose Globus turnkey solution into microservices âą Enable flexible re-composition and adaptation of services âą Support extension and enhancement of publication flows Automate SearchIdentifyDescribeTransferAuth Create folder Transfer data Get metadata Mint persistent identifier Catalog Get credentials Set ACL
- 50. Globus Search service âą Hosted, scalable service for research data discovery âą Schema agnostic âą Fine grained access control âą Plain text search âą Faceted search âą Rich query language 50
- 51. Globus Identifiers service (limited beta release) âą Issue persistent identifiers⊠⹠âŠwithin your namespace, with access control âą Identifiers have⊠â âŠlink to data â âŠlanding page â âŠvisibility â âŠchecksum â âŠextensible metadata â âŠversioning 51
- 52. Globus Automate (coming soon) âą Composition and execution service for automating research data management âą Higher level flow description language and authoring tools âą Pluggable API to integrate any actions â e.g. automated validation, metadata extraction âą Flexible invocation of actions: user or event driven
- 54. Jupyter + Globus for interactive data science at scale petrel.alcf.anl.gov materialsdatafacility.org 2PB, 80Gbps store 3.2M materials data Cooley: 290 TFLOPS Query1 Share4 Transfer2 Learn3
- 55. Genotype imputation: Wellcome Sanger
- 58. Globus PaaS developer resources Python SDK Sample Application docs.globus.org/api github.com/globus Jupyter Notebook
- 60. 8,300 active shared endpoints 70+ petabyte movers 500 PB moved 20,400 active personal endpoints 80 billion files processed 1,800 active server endpoints 94 subscribers 1 PB largest single transfer to date 99.9% availability 559 identity providers 1,923 most shared endpoints at a single institution 120,000 users Globus by the numbers
- 61. Thank you to our sponsors... U . S . D E P A R T M E N T O F ENERGY
- 62. âŠand THANK YOU, subscribers!
- 63. Globus sustainability model âą Standard Subscription â Sharing, data publication â HTTPS access â Console, usage reporting â Priority support â App integration support âą High Assurance subscription â App instance isolation â Additional authentication assurance â Audit logging â NIST 800-53, NIST 800-171 (+ BAA) âą Branded Web Site âą Premium Storage Connectors âą Alternate Identity Provider (InCommon is standard)
- 64. Support resources âą Globus documentation: docs.globus.org âą Community email list: developer-discuss@globus.org âą Helpdesk and issue escalation: support@globus.org âą Customer engagement team âą Globus professional services team â Assist with portal/gateway/app architecture and design â Develop custom applications that leverage the Globus platform â Advise on customized deployment and integration scenarios