GlobusWorld 2021 Tutorial: Building with the Globus Platform
1 like161 views
The document outlines the use of the Globus platform for data distribution and access through modern research data portals and science gateways, emphasizing best practices for design, security, and management of data permissions. It describes components and features such as guest collections, application registration, and access controls, as well as automation options using Globus Flows for efficient data management. The document also provides resources and support channels for users to engage with the Globus community.
![Data ingest with Globus Search
12
Search
Index
POST /index/{index_id}/ingest'
Search
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": "filetype",
"subject”: "https://search.api.globus.org/abc.txt",
"visible_to": ["public"],
"content": {
"metadata-schema/file#type": "file”
}
},
...
]
}](https://image.slidesharecdn.com/tut04gw21buildingportalsautomation-210518225526/85/GlobusWorld-2021-Tutorial-Building-with-the-Globus-Platform-12-320.jpg)
![Data ingest with Globus Search
13
Search
Index
POST /index/{index_id}/ingest'
Search
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": "size",
"subject": "https://search.api.globus.org/abc.txt",
"visible_to": ["urn:globus:auth:identity:46bd0f56-
e24f-11e5-a510-131bef46955c"],
"content": {
"metadata-schema/file#size": "1000000",
"metadata-schema/file#size_human": "1MB”
}
},
...
]
}
Visibility limited to Globus Auth identity
- Single user
- Globus Group
- Registered client application](https://image.slidesharecdn.com/tut04gw21buildingportalsautomation-210518225526/85/GlobusWorld-2021-Tutorial-Building-with-the-Globus-Platform-13-320.jpg)
![Data discovery with Globus Search
14
{
"@datatype": "GSearchResult",
"@version": "2017-09-01",
"count": 1,
"gmeta": [
{
"@datatype": "GMetaResult",
"@version": "2019-08-27",
"entries": [
{ ... }
],
"subject": "https://..."
}
],
"offset": 0,
"total": 1
}
GET /index/{index_id}/search?q=type%3Ahdf5
Search
Index
Simple query
Search](https://image.slidesharecdn.com/tut04gw21buildingportalsautomation-210518225526/85/GlobusWorld-2021-Tutorial-Building-with-the-Globus-Platform-14-320.jpg)
![Data discovery with Globus Search
15
POST /index/{index_id}/search
Search
Index
Complex query
{
"filters": [
{
"type": "range",
"field_name": ”pubdate",
"values": [
{
"from": "*",
"to": "2020-12-31"
}
]
}
],
"facets": [
{
"name": "Publication Date",
"field_name": "pubdate",
...
}
]
}
Search](https://image.slidesharecdn.com/tut04gw21buildingportalsautomation-210518225526/85/GlobusWorld-2021-Tutorial-Building-with-the-Globus-Platform-15-320.jpg)
GlobusWorld 2021 Tutorial: Building with the Globus Platform
- 1. Building with Globus platform services: Search and Flows Vas Vasiliadis GlobusWorld - May 13, 2021
- 2. Data distribution/access via data portal, science gateway, data commons…
- 3. Emerging best practice: Modern Research Data Portal design pattern
- 4. Canonical data access/distribution use case • Portal/science gateway to distribute data • Interface to search and discover data of interest • Asynchronous transfer to user’s system or via HTTPS (e.g. for catalogs) • Fine-grained authorization enforced on (meta)data Search and request data of interest Transfer data to destination
- 5. Common solution components • Guest collection for “staging” data • Registered application (manages permissions) • Search index with faceted queries • Data transfer, to and from shared endpoint
- 6. Relevant Globus platform features • Guest collection creation requires authentication – Cannot be completely automated – Must be a managed endpoint • Roles for management of endpoint and tasks – Access Manager role grants the right to manage permissions – Granted to other users, groups or applications
- 7. Application registration • Set desired scopes • Set callback URL • Get client ID and secret • Consents implement least privileges principle 7 Auth developers.globus.org
- 8. Data sharing permissions management • Permissions are set per folder, on a guest collection • Permissions management can be automated • For a user – Identity: user must log in with this – Email: user gets a code via email; link to their Globus Account • For a group – Group UUID: search for group to get UUID – Access governed by membership in the group • For an application – Application identity: appclientid@clients.auth.globus.org
- 9. Application concepts • Custom application that can automatically manage permissions – Can use Globus CLI • Confidential apps: use client id and secret – Ensure application is on a secure device – Set up policy for rotation of secret (limited life tokens)
- 10. Client credential grant 10 1. Authenticate with app client id and secret 2. Access Tokens Application, Science Gateway, Data Portal (Client) 3. Authenticate as app with access tokens (to manage permissions) Globus Transfer (Resource Server) Globus Auth (Authorization Server)
- 11. Data description and discovery • (Meta)data store with fine- grained visibility controls • Schema agnostic à dynamic schemas • Simple search using URL query parameters • Complex search using search request document 11 docs.globus.org/api/search Search Index Search github.com/globus/searchable-files-demo
- 12. Data ingest with Globus Search 12 Search Index POST /index/{index_id}/ingest' Search { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": "filetype", "subject”: "https://search.api.globus.org/abc.txt", "visible_to": ["public"], "content": { "metadata-schema/file#type": "file” } }, ... ] }
- 13. Data ingest with Globus Search 13 Search Index POST /index/{index_id}/ingest' Search { "ingest_type": "GMetaList", "ingest_data": { "gmeta": [ { "id": "size", "subject": "https://search.api.globus.org/abc.txt", "visible_to": ["urn:globus:auth:identity:46bd0f56- e24f-11e5-a510-131bef46955c"], "content": { "metadata-schema/file#size": "1000000", "metadata-schema/file#size_human": "1MB” } }, ... ] } Visibility limited to Globus Auth identity - Single user - Globus Group - Registered client application
- 14. Data discovery with Globus Search 14 { "@datatype": "GSearchResult", "@version": "2017-09-01", "count": 1, "gmeta": [ { "@datatype": "GMetaResult", "@version": "2019-08-27", "entries": [ { ... } ], "subject": "https://..." } ], "offset": 0, "total": 1 } GET /index/{index_id}/search?q=type%3Ahdf5 Search Index Simple query Search
- 15. Data discovery with Globus Search 15 POST /index/{index_id}/search Search Index Complex query { "filters": [ { "type": "range", "field_name": ”pubdate", "values": [ { "from": "*", "to": "2020-12-31" } ] } ], "facets": [ { "name": "Publication Date", "field_name": "pubdate", ... } ] } Search
- 16. Data Access and Sharing • Set guest collection access rule • Check authenticated user’s Group membership • Submit Transfer task 16 Groups service Transfer service GET /groups/my_groups POST /endpoint/{endpoint_id}/access POST /transfer Groups Transfer
- 18. Automation at scale using Globus Flows
- 19. Multiple ways to “automate” data management • Scripts using the CLI (+ cron?) • Globus Timer service à scheduled/recurring transfers • Your own code calling the Globus APIs (ugh!) • Globus Flows service! – Flows comprise Actions – Actions execute against an Action Provider service endpoint – Extend by using the Action Provider Toolkit action-provider-tools.readthedocs.io/en/latest
- 20. Let’s deploy and run a simple flow… Initiate a Globus transfer task to move data to a guest collection Add an access rule allowing a Group to access the data Start End
- 22. Resources • Globus API documentation: docs.globus.org/api • Helpdesk and issue escalation: support@globus.org • Mailing list: discuss@globus.org • Globus professional services team – Assist with portal/gateway/app architecture and design – Develop custom applications that leverage the Globus platform – Advise on customized deployment and integration scenarios
- 23. Join the Globus community • Access the service: app.globus.org • Create a personal endpoint: app.globus.org/file-manager/gcp • Documentation: docs.globus.org • Engage: discuss@globus.org • Subscribe: globus.org/subscriptions • Need help? support@globus.org • Follow us: @globus