Introduction to Research Automation with Globus
0 likes36 views
The document serves as an introduction to research automation, specifically through the use of Globus, which allows for the execution of research tasks with minimal human intervention. It outlines five levels of automation ranging from manual processes using a web app to fully automated systems through event triggers. Various examples, including file transfers and setting permissions, illustrate the functionality and benefits of utilizing Globus for enhanced data management and sharing.
![Globus Command Line Interface
Automation of
simple data
management tasks
Integration with
existing scripts
(job submission …)
Open source, uses
the Python SDK
$ globus
Usage: globus [OPTIONS] COMMAND [ARGS]...
Interact with Globus from the command line
All `globus` subcommands support `--help` documentation.
Use `globus login` to get started!
The documentation is also online at https://docs.globus.org/cli/
Options:
-v, --verbose Control level of output
-h, --help Show this message and exit.
-F, --format [unix|json|text] Output format for stdout. Defaults to
text
--jmespath, --jq TEXT A JMESPath expression to apply to json
output. Forces the format to be json
processed by this expression
--map-http-status TEXT Map HTTP statuses to any of these exit
codes:
0,1,50-99. e.g. "404=50,403=51"
Commands:
api Make API calls to Globus services
bookmark Manage endpoint bookmarks
cli-profile-list List all CLI profiles which have been used
collection Manage your Collections
delete Submit a delete task (asynchronous)
endpoint Manage Globus endpoint definitions
flows Interact with the Globus Flows service](https://image.slidesharecdn.com/240227rpiworkshop02automation-240229211357-d18e7286/85/Introduction-to-Research-Automation-with-Globus-9-320.jpg)
![Flow definition
21
"StartAt": "TransferFiles",
"States": {
"TransferFiles": {
"Comment": "Transfer to a guest collection",
"Type": "Action",
"ActionUrl": "https://actions.automate.globus.org/transfer/transfer",
"Parameters": {
"source_endpoint_id.$": "$.input.source.id",
"destination_endpoint_id.$": "$.input.destination.id",
"transfer_items": [
{
"source_path.$": "$.input.source.path",
"destination_path.$": "$.input.destination.path",
"recursive.$": "$.input.recursive_tx"
}
]
},
"ResultPath": "$.TransferFiles",
"WaitTime": 60,
"Next": "SetPermission",
},
"SetPermission": {
.....
"End": True
}
}
Action
Action Provider URL
Action inputs
Timeout (seconds)
Next state](https://image.slidesharecdn.com/240227rpiworkshop02automation-240229211357-d18e7286/85/Introduction-to-Research-Automation-with-Globus-20-320.jpg)
![Flow input schema
22
{
....
"properties": {
"input": {
"type": "object",
"required": [
"source",
"destination",
"recursive_tx"
],
"properties": {
"source": {
"type": "object",
"title": "Select source collection and path",
"description": ”Source collection/path (MUST end with '/')",
"format": "globus-collection",
"required": [
"id",
"path"
],
"transfer_label": {
"type": "string",
"title": "Label for Transfer Task",
"pattern": "^[a-zA-Z0-9-_, ]+$",
"maxLength": 128,
}
....
Required inputs
Custom schema
Input type
Input type](https://image.slidesharecdn.com/240227rpiworkshop02automation-240229211357-d18e7286/85/Introduction-to-Research-Automation-with-Globus-21-320.jpg)
Introduction to Research Automation with Globus
- 1. Vas Vasiliadis vas@uchicago.edu February 27, 2024 Introduction to Research Automation
- 2. What do we mean by research “automation”? Executing research tasks* reliably, at scale, with minimal (or no) human intervention when required. *data management and computation 2
- 3. Stepping into automation using Globus • Level 1: Use the web app; it’s manual, but it may be more automated than your current process J • Level 2: Semi-automated, recurring tasks • Level 3: Automation using the Globus CLI • Level 4: Automation using Globus Flows • Level 5: “Lights-out” automation using Globus Flows with event triggers
- 4. A simple, and very common, use case Transfer data to a system for sharing Transfer Set access controls for sharing data Share We’ll use this to demonstrate 5 levels of automation
- 5. Level 1 Point-and-click using the web app 6 Example: Ad hoc sharing of results with a collaborator Transfer Share
- 6. Level 2 Timer + manual setting of permissions 7 Example: Scheduled backup set to read-only access Transfer Share
- 7. Timers 8 Scheduled and/or recurring file transfers Support all transfer and sync options hpc.nih.gov/docs/globus/globus_cron.php#cron
- 8. Level 3 Parameterized CLI script 9 Transfer Share Example: On-demand data distribution from analysis
- 9. Globus Command Line Interface Automation of simple data management tasks Integration with existing scripts (job submission …) Open source, uses the Python SDK $ globus Usage: globus [OPTIONS] COMMAND [ARGS]... Interact with Globus from the command line All `globus` subcommands support `--help` documentation. Use `globus login` to get started! The documentation is also online at https://docs.globus.org/cli/ Options: -v, --verbose Control level of output -h, --help Show this message and exit. -F, --format [unix|json|text] Output format for stdout. Defaults to text --jmespath, --jq TEXT A JMESPath expression to apply to json output. Forces the format to be json processed by this expression --map-http-status TEXT Map HTTP statuses to any of these exit codes: 0,1,50-99. e.g. "404=50,403=51" Commands: api Make API calls to Globus services bookmark Manage endpoint bookmarks cli-profile-list List all CLI profiles which have been used collection Manage your Collections delete Submit a delete task (asynchronous) endpoint Manage Globus endpoint definitions flows Interact with the Globus Flows service
- 10. Transfer and share CLI commands 11 $ globus transfer > --recursive > source_collection_uuid:source_path > guest_collection_uuid:destination_path Message: The transfer has been accepted and a task has been created and queued for execution Task ID: f5eb855c-4098-11ee-8ba2-2197ca2bfedc $ globus endpoint permission create > --group $group_uuid > --permissions $permissions > guest_collection_uuid:destination_path Granting group, ............., read access to the destination directory Message: Access rule created successfully. Rule ID: 7fe723a4-413b-11ee-88f9-03dc0e0dcc45
- 11. Exercise: Run script using the Globus CLI • Log into your instance • Go to the ~/globus-tutorials directory • Run the transfer_share.sh script $ ./transfer_share.sh > --source-collection a6f165fa-aee2-4fe5-95f3-97429c28bf82 > --source-path /cli > --guest-collection fe2feb64-4ac0-4a40-ba90-94b99d06dd2c > --sharing-path /rpi/YOUR_NAME > --group-id 50b6a29c-63ac-11e4-8062-22000ab68755
- 12. Level 4 Using a Globus Flow 13 Transfer Share Example: Moving data from instrument to campus cluster for analysis
- 13. Level 4: Automation with Globus Flows • Flows Service: A platform for managed, secure, reliable task orchestration • Flows comprise Actions à invoke Globus services; extensible to support your own services • Run via web app, CLI, API, event-based triggers*
- 14. Common tasks in most instrument scenarios Transfer raw images to HPC cluster Transfer Set access controls to allow analysis Share Flows Service 2 Actions 1 :set_permission Action Provider :transfer Action Provider
- 15. Flow lifecycle 16 • Define using JSON
- 16. Flow lifecycle 17 • Define using JSON/YAML • Deploy to Flows service
- 17. Flow lifecycle 18 • Define using JSON/YAML • Deploy to Flows service • Set access policy for visibility and execution
- 18. Flow lifecycle 19 • Define using JSON/YAML • Deploy to Flows service • Set access policy for visibility and execution • Run (debug) and monitor
- 19. Flow lifecycle: Write once, run many 20 • Define using JSON/YAML • Deploy to Flows service • Set access policy for visibility and execution • Run (debug) and monitor • …and run again!
- 20. Flow definition 21 "StartAt": "TransferFiles", "States": { "TransferFiles": { "Comment": "Transfer to a guest collection", "Type": "Action", "ActionUrl": "https://actions.automate.globus.org/transfer/transfer", "Parameters": { "source_endpoint_id.$": "$.input.source.id", "destination_endpoint_id.$": "$.input.destination.id", "transfer_items": [ { "source_path.$": "$.input.source.path", "destination_path.$": "$.input.destination.path", "recursive.$": "$.input.recursive_tx" } ] }, "ResultPath": "$.TransferFiles", "WaitTime": 60, "Next": "SetPermission", }, "SetPermission": { ..... "End": True } } Action Action Provider URL Action inputs Timeout (seconds) Next state
- 21. Flow input schema 22 { .... "properties": { "input": { "type": "object", "required": [ "source", "destination", "recursive_tx" ], "properties": { "source": { "type": "object", "title": "Select source collection and path", "description": ”Source collection/path (MUST end with '/')", "format": "globus-collection", "required": [ "id", "path" ], "transfer_label": { "type": "string", "title": "Label for Transfer Task", "pattern": "^[a-zA-Z0-9-_, ]+$", "maxLength": 128, } .... Required inputs Custom schema Input type Input type
- 22. We give you a head start 23
- 23. Run a flow app.globus.org/flows (make sure you’re a member of the “Tutorial Users” group) 25 Transfer Share
- 24. Exercise: Run Globus Flow using the web app • Find “Tutorial - Transfer and Share” in flows library • Click “Start” • Confirm the source and destination collections • Change the name of target path: /rpi/YOUR_NAME • Enter a label for the flow run • Click “Start Run” • Monitor flow progress on the “Event log” tab
- 26. EC2 Instance “Instrument” Simulating an instrument flow Monitor script transfer control Access data and run analysis 0 trigger flow run set permissions 2 Globus Connect Server Globus Connect Server ALCF Eagle transfer files 1
- 28. A more interesting scenario: cryoEM Globus Flows Carbon! Correct, classify, … Transfer Transfer raw files Compute Launch analysis job Share Set access controls Transfer Move final files to repo
- 29. Globus Flows End-to-end Automation: Serial Crystallography Image processing Data capture Carbon! Check threshold Data publication Transfer Transfer raw files Transfer Move results to repo Compute Analyze images Compute Visualize Compute Gather metadata Share Set access controls Compute Launch QA job Search Ingest to index
- 31. Extending the ecosystem: Action providers 33 • Action Provider is a service endpoint – Run – Status – Cancel – Release – Resume • Action Provider Toolkit action-provider-tools.readthedocs.io compute ACLs delete identifier transfer notify ingest mkdir search ls Xtract describe web form Custom developed docs.globus.org/api/flows/hosted-action-providers
- 32. Support resources • Flows service in web app: app.globus.org/flows • Flows documentation: docs.globus.org/api/flows • Helpdesk: support@globus.org • Customer engagement team can advise on flows • Professional services team can help build flows