Building and provisioning genomics platforms on the world’s clouds
- 1. Building and provisioning genomics platforms on the world’s clouds Enis Afgan Johns Hopkins University Galaxy Project April 2016, University of Heidelberg
- 2. World’s clouds AWS AWS (coming soon) Google Compute Engine Chameleon Jetstream NeCTAR Azure
- 4. How to appropriately utilize clouds? VM Platform Service
- 5. Standalone VM Pre-configured server that is readily available. Pros Easy to build; easy to deploy Low cloud infrastructure requirements ⟶ Transferable Cons Limited capacity (compute and storage) See it in action wiki.galaxyproject.org/Cloud/Jetstream
- 6. Scalable platform Set up a virtual cluster across multiple VMs with app services. Pros Dynamically scale compute and storage Higher-level services: persistent storage, sharing, multi- application Cons Complicated build; considerable infrastructure requirements See it in action wiki.galaxyproject.org/CloudMan
- 7. Scalable platform (cont) Data analysis spans more than one application (even if that is Galaxy). Meet Genomics Virtual Lab (GVL) Pros Versatile platform built on the scalable CloudMan cluster Includes common tutorials Cons Demanding to build Calls for more customization See it in action genome.edu.au
- 8. Ready-to-use service Use cloud resources from an always-on, public service Pros Visit a URL and start computing – no setup required Cons User quotas still apply It’s still a public service: no user customization See it in action usegalaxy.org (bwa, bowtie2 – more coming)
- 9. There’s a lot of clouds out there! AWS AWS (coming soon) Google Compute Engine Chameleon Jetstream NeCTAR Azure
- 10. How to appropriately utilize many clouds? VM Platform Service Build system
- 11. Adjustable build system Automate the process of building each component Codify knowledge about the system ⟶ easier to reproduce We use Ansible as the technology of choice Compose systems from configurable and reusable roles Galaxy-Kickstarter Playbook artbio.github.io/ansible-artimed/ Galaxy-CloudMan Playbook github.com/galaxyproject/ galaxy-cloudman-playbook Use-Galaxy Playbook github.com/galaxyproject/ usegalaxy-playbook
- 12. Many clouds AND many solutions !?! launch.genome.edu.au ; use.jetstream-cloud.org ; launch.usegalaxy.org
- 13. CloudBridge (future) A Simple Cross-Cloud Python Library 1. Offer a uniform API irrespective of the underlying provider 2. Provide a set of conformance tests for all supported clouds 3. Focus on mature clouds with a required minimal set of features 4. Be as thin as possible Support for AWS and OpenStack exists; Google Cloud under development cloudbridge.readthedocs.org
- 14. CloudLaunch (future) A centralized launcher for any app and any cloud. User configurable applications and clouds; view and launch shared instances; multi-cloud dashboard view github.com/galaxyproject/cloudlaunch github.com/galaxyproject/cloudlaunch-u
- 15. CloudMan (future) Resource manager with configurable service layer • Pull away from low-level application service management • Leverage containers to supply services • Allow runtime service and configuration changes • Run on any infrastructure, including high-level services, such as ECS, or Docker API Goal: Launch a (template-based) CloudMan platform and add application services as desired from Dockerhub or similar while resource provisioning is automatically handled.
- 16. Galaxy ObjectStore (future) Allow uniform any-Galaxy computing (i.e., make Galaxy instances interchangeable and disposable) • Galaxy implements an ObjectStore interface as an abstraction to data • Leverage it to expand user data storage and allow any Galaxy to connect to a user’s bucket • Use ObjectStore for reference data (simplify builds) • Still will need to deal with the database dependency
- 17. The endgame? launch.usegalaxy.org ObjectStore CloudBridge CloudMan A P P L I C A T I O N S
- 18. Building your own cloud? Make it easy For end-users to register and get onboard (very simple auth) For deployers to interface with the cloud (adopt ‘standards’) Develop capacity and usage plans Go for monthly-reset, merit-based Allocation Units (AUs) Design for flexibility Users need more storage? Different instance types? Create champion teams Bring them onboard early to deploy target apps; give them $$$ Start with good documentation Technical but not overly detailed (look at AWS) Be open; add great, interactive support Design a training program For application developers and end users; build a community
- 19. Acknowledgments
Editor's Notes
- #21: Standards: expose API, reliable/adopted middleware,