Introduction to Globus Connect for System Administrators.pdf
- 1. Introduction to Globus Connect for System Administrators Lev Gorenstein Solutions Architect
- 2. 3 Agenda • Concepts and definitions • Prerequisites and deployment recommendations • Creating a Globus endpoint and collections • Common GCS configuration options • Monitoring and managing user activity
- 3. 4 Our focus in this session • Makes your storage accessible via Globus • Software/tools installed and managed by sysadmin • Native packaging Linux: DEB, RPM docs.globus.org/globus-connect-server
- 4. 5 A sysadmin’s view of Globus Connect Server
- 5. 6 Globus Connect Server Architecture
- 6. 7 Endpoint – A logical construct that identifies an instance of Globus Connect to the Globus service
- 7. 8 Data Transfer Node (DTN) – A physical manifestation of the endpoint More DTNs => larger physical footprint => better resilience and performance
- 8. 9 Connector – A package that implements an interface allowing the Globus service to access a specific storage system Can safely be ignored for all intents and purposes, since one never interacts directly with a “connector”
- 9. 10 Storage Gateway – An instance of a Globus connector configured to access a storage system using specified policies (valid IdPs, path restrictions, et al.)
- 10. 11 Collection – A logical construct that allows a user to access data via the Globus service (constrained by the underlying Storage Gateway) Think: a “projection” of (some part of) a storage system via the Globus service
- 11. 12 A researcher’s view of Globus Connect Server
- 14. 15 Mapped and guest collection views
- 15. 16 GCS services conceptual architecture Data Transfer Node GCS Command Line Interface GridFTP Server Globus Transfer Service GCS management requests Globus Auth Service GCS Manager authorize request using client ID/secret GCS Manager endpoint: abc.abc.data.globus.org Installation register GCS client with Globus Auth; first-timers will need to create a Project to contain the registration Define Globus resources (gateways, collections, …)
- 16. 17 Recommended deployment Source security filters (ACLs) Destination security filters (ACLs) Destination Science DMZ Source Science DMZ Source Border Router Destination Border Router Source Router Destination Router User Organization DATA CONTROL Physical Control Path Logical Control Path Physical Data Path Logical Data Path * Port 443 * Ports 50000- 51000 Data Transfer Node (DTN) Data Transfer Node (DTN) docs.globus.org/globus-connect-server/v5.4/#open-tcp-ports_section Credit: fasterdata.es.net/science-dmz/
- 17. 19 Key prerequisite: network accessibility • Yes, you must have a chat with OpSec, NetOps, … • Control channel: must be on publicly routable IP – Default: port 443; configurable – Inbound and outbound traffic from all – Can restrict to CIDR block, but severely limits functionality • Data channel: can be on private network – Default: 50000-51000 – Configurable, but strongly advise against it • But you already have Science DMZ, right? ;-) docs.globus.org/globus-connect-server/v5.4/gcsv54-restricted-firewall- policy/consequences-of-restricting-gcsv54-firewall-policy/
- 19. 21 Require a Globus subscription GCSv5 installation/configuration summary 1. Install GCS packages on data transfer node (DTN) 2. Set up the endpoint and add node(s) 3. Create a POSIX storage gateway 4. Create a mapped collection 5. Associate endpoint with a subscription 6. Create a guest collection 7. Enable browser down/upload (HTTPS access)
- 20. 22 1. Install Globus Connect Server v5 packages $ curl -LOs http://downloads.globus.org/globus-connect- server/stable/installers/repo/deb/globus-repo_latest_all.deb $ dpkg -i globus-repo_latest_all.deb $ apt-key add /usr/share/globus-repo/RPM-GPG-KEY-Globus $ apt-get update $ apt-get --assume-yes install globus-connect-server54 Already done on the demo EC2 instances
- 21. 23 Endpoint creation and node setup
- 22. 24 2. Set up endpoint and add node $ globus-connect-server endpoint setup > "My Endpoint" > --organization "My Organization" > --contact-email me@uchicago.edu > --owner me@uchicago.edu $ sudo globus-connect-server node setup Note: endpoint setup command generates deployment-key.json Safeguard this file cannot modify/restore endpoint without it Identity must be known to Globus Auth; log in and confirm prior to endpoint setup
- 23. 25 GCS clients and projects • Running endpoint setup registers a Globus Auth client • Projects are an organizing construct, necessary for access control over clients and their secrets • A project may be automatically created, if you have not previously installed GCS or registered an app with Globus Auth • View projects and Globus Auth clients: Settings Developers • If you have existing endpoints/client applications registered with Globus, endpoint setup will ask for a project ID
- 24. 26 Set up endpoint and add a DTN • Run: globus-connect-server endpoint setup ... • Run: sudo globus-connect-server node setup • Run: systemctl restart apache2 • Display endpoint details: – globus-connect-server login localhost – globus-connect-server endpoint show
- 25. 27 Our setup so far Run globus-connect-server node setup to set up additional data transfer nodes Copy deployment-key.json from original DTN
- 26. 28 Storage Gateways define a set of access policies • Authentication for local account-holders – Which identity domain(s) are acceptable? – How are identities mapped from domain(s) to local accounts? • Policy scope – Which parts of the storage system are accessible via Globus? – Which local accounts does this policy allow (or deny)? • High Assurance settings • MFA requirements
- 27. 29 Authentication for local account-holders • Primary access (via a mapped collection) requires an account on the host system* • Two-part authentication configuration: 1. Pick one or more identity domains 2. Configure the method to map the authenticated identity to an account on your system * You may allow primary users to share with others who don’t have accounts on your system
- 28. 30 Picking identity domains • User must present identity from one of the configured domains – On access attempts, linked identities will be scanned for a match – If no identity from the required domain(s), will be asked to link one • Identity domains may include… – …any organization in Globus federated list – …your institution’s identity provider trusted by Globus – …a local OpenID Connect (OIDC) server using your PAM stack
- 29. 31 Mapping identities to local accounts • Default: Strip identity domain (everything after “@”) – e.g. userX@uchicago.edu maps to local account userX – Best for campus identities w/synchronized local accounts • Use --identity-mapping option on storage gateway – Specify expression in a JSON document – Execute a custom script docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide/
- 30. 32 Create a POSIX storage gateway
- 31. 33 Creating a storage gateway • Our storage gateway will access a POSIX system – This is the only type permitted without a subscription • It will allow access to users with credentials from the example.edu (substitute your own) domain • Reauthentication will be required every 90 minutes
- 32. 34 3. Create a storage gateway $ globus-connect-server storage-gateway create posix > "My Storage Gateway" > --domain uchicago.edu > --authentication-timeout-mins 90 Allowed authentication domain Duration of user session when accessing collections via this storage gateway
- 33. 35 Our setup so far… / (user home)
- 34. 36 Create a mapped collection on the POSIX gateway
- 35. 37 Creating a collection • It will be “rooted” at the top of the filesystem (with “landing zone” in the user’s home directory) • Per our Storage Gateway: – Our collection will use the default identity mapping – Access will require authentication with an identity from the example.edu (substitute your own) domain
- 36. 38 4. Create a mapped collection $ globus-connect-server collection create > f77ff456-1f18-41d3-94a7-f3fd8858ea4d > / > "My Mapped Collection" Collections are rooted at the specified base path Can be restricted to a subset of locations below it (example later) Landing location can be adjusted (defaults to user home directory) Storage gateway ID Collection base path
- 37. 39 Our setup so far…
- 39. 41
- 41. 43 We are using the default identity mapping, so… • Create a local user account with the same name as the Globus identity you used to set up the endpoint – e.g., for me@example.edu create local account “me” – e.g., for me2@orcid.org create local account “me2” • adduser --disabled-password --gecos 'me' me • Access your mapped collection via the web app… • …and move some files, if you like
- 42. 44 Local account restrictions • Note: These only apply to mapped collections • A storage gateway’s allowed identity domains and identity mapping method determine the universe of local accounts that may access the mapped collection • You can further narrow the access universe using… --user-allow --user-deny --posix-group-allow (POSIX storage gateways only) --posix-group-deny (POSIX storage gateways only)
- 43. 45 Path restrictions • Always use the narrowest base path possible for your storage gateway(s) and collection(s) – Storage gateway base specifies where collections may be created – Collection base specifies the base directory for the collection • POSIX storage gateway – Use --restrict_paths to specify narrower read, read/write, or none access for specific paths – You provide a JSON doc that lists paths for each permission type – Note: These are absolute paths on the host system • Collection: specify narrowest base path that satisfies the need
- 45. 47 Setting path restrictions • A new storage gateway will limit access to /home – NB: No change to local permissions, only visibility via Globus • We specify the path restrictions in paths.json – See sample file in /home/adminN • Run: storage-gateway create command with the --restrict-paths option • Create a new POSIX mapped collection
- 46. 48 5. Create a restricted storage gateway, collection $ globus-connect-server storage-gateway create posix > "My Storage Gateway - Restricted" > --domain uchicago.edu > --authentication-timeout-mins 90 > --restrict-paths file:/home/adminN/paths.json $ globus-connect-server collection create > 3926bf02-6bc3-11e7-a9c6-22000bf2d287 > / > "My Mapped Collection – Restricted" Fully qualified filename containing rule(s) for restricting access to specific filesystem paths
- 47. 49 Revisit your mapped collections • Your will need to authenticate on your new (restricted access) collection, and consent • Note the access behavior differences between the two mapped collections
- 48. 51 Subscribing* your endpoint(s) • As a member of an institution’s subscription, you can subscribe your own endpoint(s) • Subscription Managers can subscribe any endpoint at the institution • Check role in your institution’s subscription to determine level of access and control app.globus.org/settings/subscriptions * Subscribed endpoints were previously called “managed”
- 49. 52 A Subscribed Endpoint • Allow guest collections enables sharing – Restrict sharing: paths, local users, local groups • Enables HTTPS access – On by default, but you can disable it • Assign additional roles for endpoint management – Administrator, Manager, Monitor
- 50. 53 Associate the endpoint with a subscription
- 51. 55 Subscribe* your endpoint • Confirm that your Globus identity is a subscription member • Run: globus-connect-server endpoint set- subscription-id • Confirm: globus-connect-server endpoint show
- 52. 56 6. Associate endpoint with a subscription $ globus-connect-server endpoint set-subscription-id DEFAULT $ globus-connect-server endpoint set-subscription-id > 39299902-6bc3-aa56-a9c6-22000bf2d287 Subscription managers can also set this via the web app Console page: app.globus.org/console (look under the Endpoints tab) Your identity may already be a member of a subscription
- 53. 57 Be identity-, role-, and permission-aware • Default: Only endpoint owner can configure an endpoint • Delegate administrator role to other sysadmins – Best practice: Delegate to a Globus group, not individuals • Check identity using s e s s i on s how • Check resource permissions on storage gateways and collections with --include-private-policies option docs.globus.org/globus-connect-server/v5.4/reference/role/
- 54. 58 7. Create a guest collection • Created by user, but administrator must enable – Set the --allow-guest-collections flag • Grants access to specific Globus users without a mapped local account • “Guest” users have the same (or more limited) permissions as the guest collection creator – Access logs show access by the collection creator* • Guest collection’s root is relative to the mapped collection’s base path * High Assurance collections log guest user identities to enable auditing
- 55. 59 Sharing restrictions • Guest collections may be created in any directory accessible by the collection, by any authorized local account • You can restrict the authorized accounts… o --sharing-user-allow --sharing-user-deny o --posix-sharing-group-allow o --posix-sharing-group-deny • …and sharing paths… o --sharing-restrict-paths (specify JSON PathRestrictions) • You can also set policies for specific user/path combinations o $ globus-connect-server sharing-policy create ... • “Sharing to” domains (guest-auth-policy) • Guest collection access time limits (--acl-expiration-mins) • Guest collection deletion on inactivity (--auto-delete-timeout)
- 56. 60 Create and access a guest collection
- 57. 61 Create and access a guest collection • Enable creation of guest collections • Access mapped collection; create directory "sharing" • Create a guest collection on sharing • Grant read access to another user or group • Authenticate and browse guest collection
- 58. 62 8. Enable web browser upload/download • Authorized users can upload, download files via a browser • Must have permissions to the collection – Collection configuration governs access – Web server is a different application separate authentication
- 60. 64 Enable HTTPS access • Run: globus-connect-server collection update – Set the --enable-https (or --disable-https) flag • Access your mapped collection • Upload a file from your laptop (and download it too!)
- 62. 66 Things to do with the management console • Monitor current transfers on your endpoints – See what’s going on at the transfer request level – Much better than watching individual file transfers • Pause (and later resume) a transfer in progress – Sends a notice to the transfer owner • Set a pause rule for current and future transfers – Ideal for maintenance mode – Notifies transfer owners, – Tasks resume when endpoint is un-paused docs.globus.org/management-console-guide/
- 63. 67 Resources • GCSv5 Guides: docs.globus.org/globus-connect-server/ • YouTube Videos: – https://www.youtube.com/watch?v=ayDW5G8mJOM – https://www.youtube.com/watch?v=8ILtsSRiML8&t – https://www.youtube.com/watch?v=86uEdOOfY7g • Globus support: support@globus.org