Globus for System Administrators
0 likes45 views
This document provides a comprehensive guide for system administrators on deploying and managing a Globus Connect Server (GCS). It details the installation process, endpoint creation, configuration options, and user management for a GCS to facilitate accessible data storage and transfers. Key features include creating storage gateways, managing authentication, and utilizing a management console for monitoring transfers.
Globus for System Administrators
- 1. Vas Vasiliadis vas@uchicago.edu February 27, 2024 Globus for System Administrators
- 2. To participate see: bit.ly/gw-tut-rpi Follow instructions under Globus for System Administrators 2
- 3. Our focus in this session • Makes your storage accessible via Globus • Software/tools installed and managed by sysadmin • Native packaging Linux: DEB, RPM docs.globus.org/globus-connect-server
- 4. Agenda • Globus Connect Server Overview • Deploying a Globus endpoint and creating collections • Common GCS configuration options • Monitoring and managing user activity 5
- 6. Globus Connect Server Components
- 7. GCS management conceptual architecture 8 Data Transfer Node GCS Command Line Interface GridFTP Server Globus Transfer Service GCS management requests Globus Auth Service GCS Manager authorize request using client ID/secret GCS Manager endpoint: abc.abc.data.globus.org Installation register GCS client with Globus Auth; first-timers will need to create a Project to contain the registration Define Globus resources (gateways, collections, …)
- 9. Key prerequisite: network accessibility • Yes, you must have a chat with OpSec, NetOps, … • Control channel: must be on publicly routable IP – Default: port 443; configurable – Inbound and outbound traffic from all – Can restrict to CIDR block, but severely limits functionality • Data channel: can be on private network – Default: 50000-51000 – Configurable, but strongly advise against it • But you already have Science DMZ, right? ;-)
- 10. Requires a Globus subscription GCSv5 installation/configuration summary 1. Install GCS packages on data transfer node (DTN) 2. Set up the endpoint and add node(s) 3. Create a POSIX storage gateway 4. Create a mapped collection 5. Associate endpoint with a subscription 6. Create a guest collection 7. Enable browser down/upload (HTTPS access) 8. Add other storage systems to the endpoint
- 11. 1. Install Globus Connect Server v5 packages $ curl -LOs http://downloads.globus.org/globus-connect- server/stable/installers/repo/deb/globus-repo_latest_all.deb $ dpkg -i globus-repo_latest_all.deb $ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus $ apt-get update $ apt-get --assume-yes install globus-connect-server54 Already done on your EC2 instances.
- 12. Endpoint creation and node setup 13
- 13. 2. Set up endpoint and add node $ globus-connect-server endpoint setup > "My Endpoint" > --organization "My Organization" > --contact-email me@uchicago.edu > --owner me@uchicago.edu $ sudo globus-connect-server node setup Note: endpoint setup command generates deployment-key.json Use this file when setting up additional data transfer nodes Identity must be known to Globus Auth; log in and confirm prior to endpoint setup
- 14. Set up endpoint and add a DTN • Access server: ssh adminN@rpiN.globusdemo.org • Switch to root: sudo su • Run: globus-connect-server endpoint setup ... – Ensure --owner is the identity you used to register the GCS • Run: globus-connect-server node setup ... • Run: systemctl restart apache2 • Display endpoint details: – globus-connect-server login localhost – globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut-rpi
- 15. GCS clients and projects • Running the endpoint setup command creates a Globus Auth client • If this is your first GCS installation, a project will also be automatically created • Projects are mostly an organizing construct, necessary for access control over clients and their secrets • View projects and Globus Auth clients: Settings à Developers • If you have existing endpoints/client applications registered with Globus, endpoint setup will ask for a project ID
- 16. Our setup so far Run globus-connect-server node setup to set up additional data transfer nodes Copy deployment-key.json from original DTN
- 17. Storage Gateways define a set of access policies • Authentication for local account-holders – Which identity domain(s) are acceptable? – How are identities mapped from domain(s) to local accounts? • Policy scope – Which parts of the storage system are accessible via Globus? – Which local accounts does this policy allow (or deny)? • High Assurance settings • MFA requirements
- 18. Authentication for local account-holders • Primary access (via a mapped collection) requires an account on the host system* • Two-part authentication configuration: 1. Pick one or more identity domains 2. Configure the method to map the authenticated identity to an account on your system * You may allow primary users to share with others who don’t have accounts on your system
- 19. Picking identity domains • User must present identity from one of the configured domains – On access attempts, linked identities will be scanned for a match – If no identity from the required domain(s), will be asked to link one • Identity domains may include… – …any organization in Globus federated list – …your institution’s identity provider trusted by Globus – …a local OpenID Connect (OIDC) server using your PAM stack
- 20. Mapping identities to local accounts • Default: Strip identity domain (everything after “@”) – e.g. userX@uchicago.edu maps to local account userX – Best for campus identities w/synchronized local accounts • Use --identity-mapping option on storage gateway – Specify expression in a JSON document – Execute a custom script docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
- 21. Create a POSIX storage gateway 22
- 22. Creating a storage gateway • Our storage gateway will access a POSIX system – This is the only type permitted without a subscription • It will allow access to users with credentials from the uchicago.edu (subsctitute your own) domain • Reauthentication will be required every 90 minutes Cheatsheet bit.ly/gw-tut-rpi
- 23. 3. Create a storage gateway $ globus-connect-server storage-gateway create posix > "My Storage Gateway" > --domain uchicago.edu > --authentication-timeout-mins 90 Allowed authentication domain Duration of user session when accessing collections via this storage gateway
- 24. Our setup so far… / (user home)
- 25. Create a mapped collection on the POSIX gateway 26
- 26. Creating a collection • Our collection will use the default identity mapping • It will be “rooted” at the user’s home directory • Access will require authentication with an identity from the uchicago.edu (substitute your own) domain Cheatsheet bit.ly/gw-tut-rpi
- 27. 4. Create a mapped collection $ globus-connect-server collection create > f77ff456-1f18-41d3-94a7-f3fd8858ea4d > / > "My Mapped Collection" Collections are rooted at the specified base path Specifying "/" as the base path sets the collection root to the local user’s home directory Storage gateway ID Collection base path
- 28. Our setup so far…
- 32. We are using the default identity mapping, so… • Create a local user account with the same name as your globusid.org (or other IdP) identity – e.g., for me@uchicago.edu create local account “me” – e.g., for me2@orcid.org create local account “me2” • adduser --disabled-password --gecos 'me' me • Access your mapped collection via the web app… • …and move some files, if you like Cheatsheet bit.ly/gw-tut-rpi
- 33. Common Collection configuration options • Restrict access: local users, local groups • Restrict sharing: paths, local users, local groups • Allow guest collections à enables sharing • Enable HTTPS access • Force data channel encryption
- 34. Local account restrictions • Note: These only apply to mapped collections • A storage gateway’s allowed identity domains and identity mapping method determine the universe of local accounts that may access the mapped collection • You can further narrow the access universe using… --user-allow --user-deny --posix-group-allow (POSIX storage gateways only) --posix-group-deny (POSIX storage gateways only)
- 35. Path restrictions • Always use the narrowest base path possible for your storage gateway(s) and collection(s) – Storage gateway base specifies where collections may be created – Collection base specifies the base directory for the collection • POSIX storage gateway – Use --restrict_paths to specify narrower read, read/write, or none access for specific paths – You provide a JSON doc that lists paths for each permission type – Note: These are absolute paths on the host system • Collection: specify narrowest base path that satisfies the need
- 36. Restrict collection access to filesystem 38
- 37. Setting path restrictions • A new storage gateway will limit access to /home – NB: No change to local permissions, only visibility via Globus • We specify the path restrictions in paths.json – This file is in your admin user’s home directory • Run: storage-gateway create command with the --restrict-paths option • Create a new POSIX mapped collection Cheatsheet bit.ly/gw-tut-rpi
- 38. 5. Create a restricted storage gateway, collection $ globus-connect-server storage-gateway create posix > "My Storage Gateway - Restricted" > --domain uchicago.edu > --authentication-timeout-mins 90 > --restrict-paths file:/home/adminN/paths.json $ globus-connect-server collection create > 3926bf02-6bc3-11e7-a9c6-22000bf2d287 > / > "My Mapped Collection – Restricted" Fully qualified filename containing rule(s) for restricting access to specific filesystem paths
- 39. Revisit your mapped collections • Your will need to authenticate on your new (restricted access) collection, and consent • Note the access behavior differences between the two mapped collections
- 40. Subscriptions and Endpoint Roles • Subscription(s) configured for your institution • Multiple Subscription Managers per subscription • Subscription Manager associates an endpoint with a subscription – Results in a “subscribed” endpoint • Assign additional roles for endpoint management – Administrator, Manager, Monitor
- 41. Converting endpoint(s) to “subscribed” • As a member of an institution’s subscription, you can subscribe your own endpoint(s) • Subscription Managers can subscribe any endpoint at the institution • Check role in your institution’s subscription to determine level of access and control • But, not yet … J
- 42. Associate the endpoint with a subscription 44
- 43. Convert your endpoint to “subscribed” • Confirm that your Globus identity is a subscription manager à we’ll do this for you now • Run globus-connect-server endpoint set- subscription-id • Confirm: globus-connect-server endpoint show Cheatsheet bit.ly/gw-tut-rpi
- 44. 6. Associate endpoint with a subscription $ globus-connect-server endpoint set-subscription-id DEFAULT $ globus-connect-server endpoint set-subscription-id > 39299902-6bc3-aa56-a9c6-22000bf2d287 Subscription managers can also set this via the web app Console page: app.globus.org/console (look under the Endpoints tab) Your identity may already be a subscription manager on a subscription
- 45. Be identity-, role-, and permission-aware • Default: Only endpoint owner can configure an endpoint • Delegate administrator role to other sysadmins – Best practice: Delegate to a Globus group, not individuals • Check identity using the session command • Check resource permissions on storage gateways and collections with --include-private-policies option docs.globus.org/globus-connect-server/v5.4/reference/role/
- 46. 7. Create a guest collection • Created by user, not endpoint administrator • Grants access to specific Globus users without a mapped local account • “Guest” users have the same (or more limited) permissions as the guest collection creator – Access logs show access by the collection creator* • Guest collection’s root is relative to the mapped collection’s base path * High Assurance collections log guest user identities to enable auditing
- 47. Sharing restrictions • Guest collections may be created in any directory accessible by the collection, by any authorized local account • You can restrict the authorized accounts… o --sharing-user-allow --sharing-user-deny o --posix-sharing-group-allow o --posix-sharing-group-deny • …and sharing paths… o --sharing-restrict-paths (specify JSON PathRestrictions) • You can also set policies for specific user/path combinations o $ globus-connect-server sharing-policy create ...
- 48. Create and access a guest collection 51
- 49. Create and access a guest collection • Enable creation of guest collections • Run: globus-connect-server collection update • Access the mapped collection; create /projects • Create a guest collection on the /projects directory • Grant read access to the “Tutorial Users” group • Authenticate and browse guest collection
- 50. 8. Enable web browser upload/download • Authorized users can upload, download files via a browser • Must have permissions to the collection – Collection configuration governs access – Web server is a different application (separate authentication)
- 52. Enable HTTPS access • Run: globus-connect-server collection update • Access your mapped collection • Upload a file from your laptop (and download it too!) Cheatsheet bit.ly/gw-tut-rpi
- 54. Things to do with the management console • Monitor current transfers on your endpoints – See what’s going on at the transfer request level – Much better than watching individual file transfers • Pause (and later resume) a transfer in progress – Sends a notice to the transfer owner • Set a pause rule for current and future transfers – Ideal for maintenance mode – Notifies transfer owners, – Tasks resume when endpoint is un-paused docs.globus.org/management-console-guide/
- 55. Resources • GCSv5 Guides: docs.globus.org/globus-connect-server/ • Migration: docs.globus.org/globus-connect- server/migrating-to-v5.4/ • Globus support: support@globus.org 58