Climb stateoftheartintro
- 1. Microbial Bioinformatics in the cloud Introducing CLIMB Dr Tom Connor Cardiff University Biological Cloud Computing Workshop www.climb.ac.uk @tomrconnor ; @mrcclimb
- 2. Overview • Background • View from a newly minted academic Bioinformatician at a “regular” University – Bioinformatics needs and challenges • Introducing CLIMB
- 3. Big Data Wave 1 Wave 2 Wave 3 2005- 09 1989- 97 2003- 07 1992- 2002 1993-98 1975-86 1937-611966-71 1967-89 1969-73 1969-81 1981-85 1974 1986-87 1969-73 Adapted from Mutreja, Kim, Thomson, Connor et al, Nature, 2011 Population genomics; using genomics to reconstruct the global spread of pathogens Bioinformatics; developing new approaches to analyse massive datasets From Marttinen, Hanage, Croucher, Connor, Harris, Bentley and Corander, Nucleic Acids Res. 2011 From Cheng, Connor, Sirén, Aanensen, Corander, MBE, 2013 Grand challenges; fighting antimicrobial resistance From Fookes et al, PLoS Pathogens 2011 From Reuter, Connor et al, PNAS 2014 From Okoro, Kingsley, Connor et al. Nature Genetics 2012 From He et al, Nature Genetics 2013 From Dziva, Hauser, Connor et al, I&I, 2013 Pathogen genomics: understanding how pathogens evolve
- 4. About Me • 2006-2010 - PhD at Imperial; Population Genetics / Molecular Epidemiology of bacterial pathogens • 2010-2012 - Post-Doc, Wellcome Trust Sanger Institute; Pathogen Genomics • 2012 – present Lecturer then Senior Lecturer, Cardiff University Hanage, Fraser, Tang, Connor, Corander, Science, 2009
- 5. Building bioinformatics capacity • When I arrived at Cardiff, I had the joy of working out how I was going to do my research in a new place • How to get scripts/software installed • Where to install scripts/software • And I had to do this mostly on my own • Not an unusual story
- 6. Key Challenges • Infrastructure – Storage – HTC capacity • Portability of software • Portability of datasets • But, we do have the University Supercomputer….
- 7. Advanced Research Computing @ Cardiff (ARCCA) • 2048 core HPC cluster • Second hand ~868 Westmere core “HTC” partition • 8 ‘large memory’ 128GB nodes • Lustre file system (scratch, nominally unlimited, but 50TB total space initially) • NFS /home mount (50GB maximum quota) • Freely available to University Researchers
- 8. …. So the first thing I did was to buy myself some servers • Large HPC clusters often don’t meet our needs • Bioinformaticians aren’t the ideal HPC users – Disruptive software needs – Disruptive usage patterns – Disruptive storage needs • Setting up our “own” system seems the most intuitive way to ensure that you have something that works
- 9. Biologists often end up working in silos • As a discipline we have probably been taught to think in terms of ‘labs’, ‘groups’ and ‘experiments’ being wet work • We build capacity and teams locally, and those are the resources that we use every day • For bioinformaticians this means we are likely to develop our solutions locally first, building a local group and local capacity • Our software, data set storage, LIMS etc are usually bespoke • Because our software/data is locally stored/setup – it is often less portable than wet lab methods / approaches • Bioinformaticians should be working differently
- 10. Key Challenges Overall • We need systems that allow us to rapidly and easily share complete systems, from the perspective of both novice users and experienced developers • We need to develop systems to share complete datasets, rather than forcing users to install loads of bits of software to reconstitute the development environment we used, or forcing us to become proper software developers • We need to lower the barrier to access for for research scientists with a limited understanding of UNIX/Computer Science • We need a system that allows us to train users on systems that they will then be able to use when they go home • We need to understand the needs of individual fields • We need to integrate activities across these needs, to avoid reinventing the wheel
- 11. The cloud • All of the infrastructure issues are much easier when tackled at scale • This concept led to Amazon, followed by others, offering Cloud services • A cloud infrastructure provides a mechanism to share systems/software and data, at scale • Let someone else do the admin etc, and all you have to worry about is running the software
- 12. Why not use a commercial cloud? • We often want lots of RAM – Amazon max flavour size is ~250GB • Prices are high ($1200/month for ~250GB RAM flavour) • Storage costs also high – 1TB on Amazon S3 costs $30/month (our current costs are £7/month) • Additionally Amazon isn’t designed to facilitate sharing of data etc between different people who have VMs • There are possible issues around T&C’s, governance etc • Even if we overcome these, often these services are too hard for novice users to make use of
- 13. Needs • Core infrastructure now to take advantage of new technologies • Systems to easily share data • Repositories that can make tools/methods/data available rapidly and easily • Better use of the existing RCUK/University server estate • To change the view of Biologists about working in silos
- 14. Introducing the CLoud Infrastructure for Microbial Bioinformatics (CLIMB) • MRC funded project to develop Cloud Infrastructure for microbial bioinformatics • ~£4M of hardware, capable of supporting >1000 individual virtual servers • Providing a core, national cyberinfrastructure for Microbial Bioinformatics
- 15. The CLIMB Consortium Are • Professor Mark Pallen (Warwick) and Dr Sam Sheppard (Swansea) – Joint PIs • Professor Mark Achtman (Warwick), Professor SteveBusby FRS (Birmingham), Dr Tom Connor (Cardiff)*, Professor Tim Walsh (Cardiff), Dr Robin Howe (Public Health Wales) – Co-Is • Dr Nick Loman (Birmingham)* and Dr Chris Quince (Warwick) ; MRC Research Fellows * Principal bioinformaticians architecting and designing the system
- 16. The CLoud Infrastructure for Microbial Bioinformatics (climb.ac.uk) • We are creating A one stop shop for Microbial Bioinformatics – Public/private cloud for use by UK academics – Standardised cloud images that implement key pipelines – Storage repository for data/images that are made available online and within our system, anywhere (‘eduroam for microbial genomics’) • We will provide access to other databases from within the system • As well as providing a place to support orphan databases and tools
- 17. System Outline • 4 sites • Connected over Janet • Different sizes of VM available; personal, standard, large memory, huge memory • Able to support >1,000 VMs simultaneously (1:1 vCPUs/vRAM : CPUs/RAM) • 7-8PB of object storage across 4 sites (~2-3PB usable with erasure coding) • 4-500TB of local high performance storage per site • A single system, with common log in, and between site data replication • System has been designed to enable the addition of extra nodes / Universities
- 18. CLIMB Overview • 4 sites, running OpenStack • Hardware procured in a two stage process • IBM/OCF provided compute, Dell/redhat provided storage • Networks provided by Brocade • Are defining a reference architecture to enable other sites to trivially be added
- 19. Hardware (per site) • 2 router/firewalls (capable of routing >80Gb each • 3 Controllers • 21x 64 vCore, 512GB RAM nodes • 3x 192 vCore, 3TB RAM nodes • ~500TB GPFS (local) – 4 controllers – Infiniband, with 10Gb failover • ~2TB Ceph (shared) – 27x 64TB nodes/site – Cross site replication – 10Gb Backbone
- 20. Overview – 4 sites, (virtually) identical hardware Each site is connected to the others over VPN tunnels. Sites can be easily added. System can use free router software and commodity hardware, pay for-software or dedicated router/firewalls Our intention is for the system to be presented to users as a single system, with single login, via Shibboleth. We are currently working on that bit A single system makes it easy(er) to share methods and data! External clouds External databases External clouds External databases
- 21. Flavours • User configurable, with standard flavours • Regular; up to 8 vCPUs, 64GB RAM • xlarge; up to 16 vCPUs, 256GB RAM • Huge; up to 192 vCPUs, 3TB RAM • System also supports a scalable virtual cluster (large embarrassingly parallel projects) – 2+ nodes with 2+ vCPUs, 2-4GB RAM/vCPU • Also provides for Long Term Hosting (for orphan datasets/tools)
- 22. Access • Microbial researchers will be able to access the system through one of two ways – Externally, via federated access system, login via .ac.uk user login in first instance, later (hopefully) open to anyone who uses shibboleth – Internally, via user accounts setup by consortium for collaborators • Researchers will be able to provision up a set number of VMs
- 23. Where are we now? • Computational hardware was procured by March 2015 (~6 month process) • Ahead of schedule - system is now online and in use for research • Adopting two models for access – Access for registered users to core OpenStack system online now – “version 1.0” system providing universal access to predefined images starting with the GVL – Autumn 2015
- 24. VMs are already up
- 25. Users are already using CLIMB to do research
- 26. Challenges • Future Planning (CLIMB will run for 5 years, then what?) • Cross-Cloud Integration • Not reinventing the wheel • Standardising software stacks for .ac.uk clouds • Being able to embrace new technologies • Meeting cloud development needs
- 27. The Sequencing Iceberg All of the sequencing platforms available now make producing large genomics datasets relatively cheap and easy However, the major costs and difficulties do not lie with the generation of data, they lie with the pre-requisites for storing and analysing that data Informatics expertise Storage availability Appropriate HTC capacity These are interlinked, and expensive
- 28. Iceberg breaking with the Cloud? • It is a mechanism for sharing servers – Clouds remove the need for hardware maintenance and support – Storage, compute, networking are most expensive when bought one by one; building a large system represents better value for money • Sharing servers means you can have standardised systems, simplifying the process of installing and maintaining software – It provides a mechanism for software/data reuse as well as sharing – Also makes training easier; you can use the system you trained on, once you get back home • Sharing servers also makes training easier; you can use the system you trained on, once you get back home
- 29. CLIMB Next Steps – and future needs • New images/analytics tools (GVL!) • Integration of datasets • Expanding our userbase • Collaboration with other cloud services • Integrating with databases • Integrating with other clouds • Developing new sites • Developing the system to meet the needs of our users • Developing policy • Defining and developing security policy • Developing/setting up federated access • Possibly looking at capacity to burst out or accept bursts from other resources • Developing our training programme and outreach
- 30. Cloud Infrastructure for Microbial Bioinformatics • A multi site system to provide a one-stop-bioinformatics-shop, designed specifically to support Microbial researchers • For both Bioinformaticians and wet lab scientists • Combines hardware with training • Free, simple interface, easy to use • Common login • Easy data and method sharing • Already have multiple users from across UK academia and healthcare
- 31. The CLIMB Consortium Are • Professor Mark Pallen (Warwick) and Dr Sam Sheppard (Swansea) – Joint PIs • Professor Mark Achtman (Warwick), Professor SteveBusby FRS (Birmingham), Dr Tom Connor (Cardiff)*, Professor Tim Walsh (Cardiff), Dr Robin Howe (Public Health Wales) – Co-Is • Dr Nick Loman (Birmingham)* and Dr Chris Quince (Warwick) ; MRC Research Fellows * Principal bioinformaticians architecting and designing the system
- 32. CLoud Infrastructure for Microbial Bioinformatics (CLIMB) • MRC funded project to develop Cloud Infrastructure for microbial bioinformatics • ~£4M of hardware, capable of supporting >1000 individual virtual servers • Amazon/Google cloud for Academics