Easy Object Storage Import/Export Using the S3 Connector on Jetstream
1 like236 views
The document discusses easy object storage import/export using the S3 storage connector within Jetstream, a cloud platform for the research community funded by the National Science Foundation. It details how researchers can utilize Globus for transferring data into Jetstream's object storage, which is accessible via various APIs and is AWS S3 compatible. The setup process for integrating these systems and managing credentials is outlined, with an emphasis on making data transfers easy, fast, and reliable.
Easy Object Storage Import/Export Using the S3 Connector on Jetstream
- 1. Easy Object Storage Import/Export Using the S3 Storage Connector Lee Liming (LLiming@uchicago.edu) University of Chicago funded by the National Science Foundation Award #ACI-1445604
- 2. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Cloud, specialized for the research community Researchers browse VM images saved by other researchers – Web interface provided by CyVerse – Share fully-installed systems with applications ready to go – OpenStack API/CLI access for advanced users Images can be launched on VMs scaled for research – 1-44 vCPUs – 2-120 GB RAM – 8-480GB disk Use is allocated by NSF/XSEDE
- 3. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Jetstream’s object storage (Swift/Ceph) • 1.25 PB usable storage (If all of JS was object store, no VMs, volumes, etc. Of course, it’s not…) • Access via Swift APIs (and any tools that use them) – Horizon web app – OpenStack CLI • AWS S3-compatible • A place to learn & evaluate before the commercial cloud
- 4. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Research data starts & lives at home • Campus systems or research facilities/labs – Data often comes from facilities/labs/observatories – Results are often needed elsewhere – Publishing requirements are becoming important • Very often, available via Globus • So…how do we get it from there into Jetstream’s object store for use via Ceph, Swift, S3, etc? – Must be: easy, fast, reliable • (Jetstream to “next step location” is also valuable…)
- 5. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Fire-and-forget transfers into Jetstream buckets
- 6. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ So how does this work? • Globus Connect Server (the server version of Globus) offers storage connectors to subscribers, including an AWS S3 connector. • The AWS S3 connector works equally well with OpenStack Swift’s S3 dialect.
- 7. Setting it up 1. Create a VM w/floating IP. 2. Install GCSv5.2 w/S3 connector. 3. Create an S3 storage gateway requiring an XSEDE ID. 4. Use Globus web app to create a collection and supply S3-style credentials. 5. Use the collection to transfer data in or out. 6. Shelve VM until needed again.
- 8. Credential management When the GCSv5 server admin does this… …it allows users to do this.
- 9. funded by the National Science Foundation Award #ACI-1445604 http://jetstream-cloud.org/ Load dataset from EBI into a Jetstream bucket