The Outlook is Cloudy
Download as PPTX, PDF0 likes345 views
The document discusses building large-scale clouds and using utility infrastructure. It summarizes some large-scale projects using a cloud approach, including ones in financial services and digital media. It outlines benefits of the cloud approach like lower costs and flexibility. However, it also discusses issues with using clouds like unpredictable performance, security concerns, and the need for staff with diverse cloud skills. The conclusion is that organizations should not adopt clouds half-heartedly but thoughtfully integrate owned and commercial services to meet user needs.
The Outlook is Cloudy
- 1. The Outlook isCloudyBenefits, pitfalls and issues in building large-scale cloudsTerry Harmer12 May 2011EduServ 2011 Symposium1http://www.besc.ac.uk
- 2. What do I do?Technical Directorof Belfast e-ScienceDevelop project ideas for digital economy applicationsForm consortia to bid for funding usually write the project funding proposalsfunding from EPSRC, TSB, LDA and private companiesLead Technical architect for projectsProject Manager … also do software developmentI propose, design, manage and (help) build large-scale service centric applications.These projects are (and are increasingly) cloud based using utility infrastructure consisting of owned and multiple utility vendors.12 May 2011EduServ 2011 Symposium2
- 3. Talk OutlineObjectiveTo present some large-scale projects that are in or were in field deployment with established user groups
- 4. Dynamic and utility focused
- 5. Why this approach and what advantages has this approach given us.
- 6. Some technology that we use.
- 7. Issues, advantages, problems, pitfalls…OrganisationBeSC? How we started with clouds Project OverviewsFinancial servicesDigital mediaIssues12 May 2011EduServ 2011 Symposium3Cloud – Hype or Reality? For us this has been part of how we been doing things for a number of years.
- 8. Belfast e-Science Centre? Belfast e-Science was established in 2002 with funding from EPSRC and the DTI under the UK e-Science programme.EPSRC Platform Award funded and by TSB, EPSRC, INI, MoD, QinetiQBeSC is entirely self funding(and has been since 2002)Don’t really use shared resources within a University infrastructureHave close connections with companies, commercial users and organisationsWe have the attitude and tend to operate like a small R&D companyHave a tight budget and (perhaps too) big ambitions. The accidental decision to focus on commercial / industrial applications was driven by the challenges they offered.The challenges made us uniqueWe met a bunch of people who got the idea!But they wanted to do large-scale and practical examples.12 May 2011EduServ 2011 Symposium4
- 9. BeSC Infrastructure – Shared ServicesEmail has been hosted by a provider (Jan 2007)Project calendars, email lists and chat rooms are all hosted by utility providers (besc.ac.uk)All of our project shared services have migrated to utility resources (Autumn 2007 onward).…projects have progressively moved to utility cloud servicesUse a range of commercial providersTurning off internal kit that we own and shared with our commercial partners
- 10. Infrastructure circa2005200420032002BT DCTHNBBC12 May 2011EduServ 2011 Symposium6
- 11. Infrastructure 200712 May 2011EduServ 2011 Symposium7
- 12. Financial Services DRMirrorCapability CloudCapability CloudServer FarmJobManagementFuturesCommoditiesBranchesUsers12 May 2011EduServ 2011 Symposium8
- 13. Physical Resource View (2006) 12 May 2011EduServ 2011 Symposium9
- 14. Digital mediaWorking in the evolving on-demand media environmentStarted pre- iPlayer and YouTube!Concern early was on better resource utilisation in and expensive and highly dynamic environment.Early model of pooled resourcesMost recently in on-demand media infrastructuresProject PRISM with BBC/QinetiQ/BT (completed 2009)Supported game console to Phone to set-top box access.Currently supporting film and on-demand services.StatsBBC archive ~52PBiPlayer 7PB of data transfer per month12 May 2011EduServ 2011 Symposium10
- 15. A Circuit-based Infrastructure (2002)12 May 2011EduServ 2011 Symposium11
- 16. BBC Broadcast Resources (slide from 2004)12 May 2011EduServ 2011 Symposium12
- 17. Mobile Non-geographic services (slide from 2005)12 May 2011EduServ 2011 Symposium13
- 18. MediaEconomy12 May 2011EduServ 2011 Symposium14
- 19. Content CloudInfrastructure needs to scale rapidly to reflect demand12 May 2011EduServ 2011 Symposium15
- 20. A Dynamic Utility Resource Cloud (2007)12 May 2011EduServ 2011 Symposium16
- 21. Infrastructure SummaryDynamic collections of servicesManaging real user groupsServices scale and contract to established SLAsWe attempt to keep our deployed infrastructure lowCheaper … and… a small attack surfaceOur infrastructure is a mix of owned and utility infrastructureincreasingly the utility part is the majorityBuying capacity and storage on demand is our norm.Play the provider market…12 May 2011EduServ 2011 Symposium17
- 22. Issues - GeneralUtility resource market is immatureThe offerings can be difficult to compareno standard unit of compute/storagesimply specifying 1 GiBis pretty meaningless, need to talk about uptime, chance of data loss, etcPrices will be dependant on the user usage patternWhat you get and what you can buy varies widelySome attempts at customer lock-in to providersMulti-provider clouds can be (relatively) expensiveNot being multi-provider can be expensive in reliabilityProvider APIs and features constantly changing.No standard API New services and providers appearing. APIs not very well documentedWeak SLAs from providersbest SLAs guarantee 100% uptime and refund when there's downtime. There's always downtime. The cost of your downtime will be much higher than what you are paying for the resources. 12 May 2011EduServ 2011 Symposium18
- 23. Issues - TechnicalMachine performance unpredictable. CPU features especially unpredictable and can make a big difference to compute-heavy tasks e.g. we are heavy video transcoding users.individual instances can be unreliable (hosts DO crash)Bandwidth unpredictable and can be costlyrelative to JANET…more reasonable relative to real commercial ratesRequired to manage OS imagesproliferation of images; using anything but vendor images requires trust in creator. nobody has a trust framework-have to trust that usercreating own images (or using other peoples) means more machines to keep up to date!12 May 2011EduServ 2011 Symposium19
- 24. Issues - Technical/developmentYou need significant metadata on your applicationsOS versions, Software stack, security configuration, operational behaviour, versions and compatibilityWhat you are prepared to pay and how to model We develop this metadata as part of our software development processAs software is developed and integratedDevelop, manage, test costing models and scenarios for deploymentYour applications need to be developed to cope with and recover from failureMTTR rather than MTTF
- 25. Issues - Securitylow latency to other consumers' boxes decreases attacker cost and time to perform timing attacksnefarious, rich attackers can get on your box and slow you down or potentially compromise key generationSee http://people.csail.mit.edu/tromer/papers/cloudsec.pdfDDoS on cloud providers can be very damaging to everyone in it; Larger providers just increase the cost of the attack but reward is also high (see http://www.theregister.co.uk/2009/10/05/amazon_bitbucket_outage/ )No (meaningful) security QoSpost-attack analysis challenging - in many clouds you cannot inspect a disk to see logs without starting up machinePotential data security issues: who has access to physical boxes? e.g. Amazon recommends all data on disks be encrypted12 May 2011EduServ 2011 Symposium21
- 26. Issues - StaffUtility cloud approaches requireStaff with more rounded understanding of end-to-end deploymentUnderstand software developmentUnderstand and can model operational behaviourUnderstand securityDeploymentIn service managementFewer staff but ones that are better qualified…and harder to find…and harder to keep…and more expensive12 May 2011EduServ 2011 Symposium22
- 27. Don’t go cloudHalf-heartedlyPicking up your server room and placing it in a providerHere is my cloud kit…I have a cloudwithout knowing why you are doing it and what you want to achievewithout assessing the process and organisational changes you will need to make it work without knowing your applications and their behaviourbecause you think you will save money quicklyYou probably won’t initially…and think in terms of resourcesClouds of services provide a better model of useIntegrate owned and commercial services to meet user needsand adopt a single vendor or provider approachSingle point of failureyour technical roadmap will be controlled by someone elseAcademic applications have different characteristics to commercial applications12 May 2011EduServ 2011 Symposium23
- 28. Headline AdvantagesYou own what you need to own for as long as you need to own it and it can be configured for your needs.Focus on application rather on the supporting resourcesRequires applications to address resilience and surviving failureUse and be part of a marketplace of application servicesCan enable the application to remain immune from underlying technology changesDevelop an infrastructure that suits the application we are deploying.The cost of ownership is pretty low.R&D organisation we can punch above our small size and relatively small budget.Reach out to real user groups Experiment with great flexibility….. Unconstrained by (often entirely justified) corporate/academic infrastructure procedures.Security and procedure appropriate for the application12 May 2011EduServ 2011 Symposium24
- 29. Thank you12 May 2011EduServ 2011 Symposium25