My Other Computer is a Data Center (2010 v21)
Download as PPTX, PDF0 likes1,285 views
The document provides an overview of cloud computing, detailing its various types such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). It highlights the evolution of data centers, their operational costs, and the significance of virtualization and programming models for managing cloud resources. It also discusses the role of the Open Cloud Consortium in promoting cloud standards and interoperability among cloud computing platforms.
My Other Computer is a Data Center (2010 v21)
- 1. An Overview of Cloud Computing:My Other Computer is a Data CenterRobert GrossmanOpen Cloud ConsortiumJanuary 7, 2010
- 2. Part 1What is a Cloud?2
- 3. What is a Cloud?3Software as a Service (SaaS)
- 4. What Else is a Cloud?4Platform as a Service (PaaS)
- 5. Is Anything Else a Cloud?5Infrastructure as a Service (IaaS)
- 6. Are There Other Types of Clouds?6ad targeting Large Data Cloud Services
- 8. Idea Dates Back to the 1960s8AppAppAppCMSCMSMVSIBM VM/370IBM MainframeNative (Full) VirtualizationExamples: Vmware ESXVirtualization first widely deployed with IBM VM/370.
- 9. What Do You Optimize?Goal: Minimize latency and control heat.Goal: Maximize data (with matching compute) and control cost.
- 10. 10Scale is new
- 11. Elastic, Usage Based Pricing Is New11costs the same as1 computer in a rack for 120 hours120 computers in three racks for 1 hour Elastic, usage based pricing turns capex into opex.
- 12. Clouds can be used to manage surges in computing needs.Simplicity Offered By the Cloud is New12+.. and you have a computer ready to work.A new programmer can develop a program to process a container full of data with less than day of training using MapReduce.
- 13. 13
- 14. What Resource is Managed?Scarce processors wait for dataManage cycleswait for an opening in the queuescatter the data to the processorsand gather the resultsPersistent data wait for queriesManage datapersistent data waits for queriescomputation done locallyresults returnedSupercomputer Center Model Data CenterModel
- 15. Part 2. Data Centers as the Unit of ComputingCloud computing is at the top of the Gartner hype cycle.“Cloud computing has become the center of investment and innovation.”Nicholas Carr, 2009 IDC Directions15
- 18. Transition Taking PlaceA hand full of players are building multiple data centers a year and improving with each one.This includes Google, Microsoft, Yahoo, …A data center today costs $200 M – $400+ MBerkeley RAD Report points out analogy with semiconductor industry as companies stopped building their own Fabs and starting leasing Fabs from others as Fabs approached $1B 18
- 19. Which is the Operating System?19……VM 1VM 5VM 50,000VM 1Data Center Operating SystemHyperviserworkstationdata center
- 20. How Do You Program A Data Center?20
- 21. Some Programming Models for Data CentersOperations over data center of disksMapReduce (“string-based”)User-Defined Functions (UDFs) over data centerSQL and Quasi-SQL over data centerData analysis / statistics over data centerOperations over data center of memoryGrep over distributed memoryUDFs over distributed memorySQL and Quasi-SQL over distributed memoryData analysis / statistics over distributed memory
- 22. Part 3.Open Cloud Consortium
- 23. U.S. 501(3)(c) not-for-profit corporationSupports the development of standards and interoperability frameworks.Supports reference implementations for cloud computing. Manages testbeds: Open Cloud Testbed, IntercloudTestbed, Open Science Data CloudDevelops benchmarks.23www.opencloudconsortium.org
- 24. OCC MembersCompanies: Aerospace, Booz Allen Hamilton, Cisco, InfoBlox, Open Data Group, Raytheon, YahooUniversities: CalIT2, Johns Hopkins, Northwestern, University of Illinois at Chicago, University of ChicagoGovernment agencies: NASAOrganizations: Sector Project24
- 26. Sector/Sphere
- 27. Thrift
- 28. KVM VMs
- 30. IntercloudTestbedPlatform as a Service
- 32. Data & Storage as a ServiceLarge Data Cloud Interoperability FrameworkWorking with Infrastructure 2.0 Working GroupSNIA Cloud Data Management Interface (CDMI)Dynamic infrastructure service linking IaaS and DaaSWorking with Infrastructure 2.0 Working GroupInfrastructure as a ServiceVirtual Data Centers (VDC)Virtual Networks (VN)Virtual Machines (VM)Physical ResourcesDynamic infrastructure service naming and linking entities in the IaaS layersOpen Cloud Computing Interface (OCCI)Open Virtualization Format (OVF)
- 33. Open Science Data Cloudsky cloudPlanning to work with 5 international partners (all connected with 10 Gbps networks).biocloud27
- 34. MalStone (OCC-Developed Benchmark)Sector/Sphere 1.20, Hadoop 0.18.3 with no replication on Phase 1 of Open Cloud Testbed in a single rack. Data consisted of 20 nodes with 500 million 100-byte records / node.
- 35. Some Lessons Learned (So Far)Python over Hadoop Distributed File System surprisingly powerful.Tuning Hadoop can be a large (unacknowledged) cost. Performance of a cloud computation can be significantly impacted by just 1 or 2 nodes that are a bit slower.Wide area clouds can be practical in some cases.29
- 36. Part 4. Sector30http://sector.sourceforge.net
- 37. Sector OverviewSector is fastAs measured by MalStone & TerasortSector is easy to programSupports UDFs, MapReduce & Python over streamsSector does not require extensive tuning.Sector is secureA HIPAA compliant Sector cloud is being set upSector is reliableSector v1.24 supports multiple master node servers31
- 38. Google’s Large Data CloudCompute ServicesData ServicesStorage Services32ApplicationsGoogle’s MapReduceGoogle’s BigTableGoogle File System (GFS)Google’s Stack
- 39. Hadoop’s Large Data CloudCompute ServicesStorage Services33ApplicationsHadoop’sMapReduceData ServicesHadoop Distributed File System (HDFS)Hadoop’s Stack
- 40. Sector’s Large Data Cloud34ApplicationsCompute ServicesSphere’s UDFsData ServicesSector’s Distributed File System (SDFS)Storage ServicesUDP-based Data Transport Protocol (UDT)Routing & Transport ServicesSector’s Stack
- 41. Generalization: Apply User Defined Functions (UDF) to Files in Storage Cloudmap/shufflereduce35UDFUDF
- 42. Hadoopvs Sector36Source: Gu and Grossman, Sector and Sphere, Phil. Trans. Royal Society A, 2009.
- 43. Terasort - Sector vsHadoop PerformanceSector/Sphere 1.24a, Hadoop 0.20.1 with no replication on Phase 2 of Open Cloud Testbed with co-located racks.
- 44. Sector ApplicationsDistributing the 15 TB Sloan Digital Sky Survey to astronomers around the world (joint with JHU, 2005)Managing and analyzing high throughput sequence data (Cistrack, University of Chicago, Cistrack, 2007).Detecting emergent behavior in distributed network data (Angle, won SC 07 Analytics Challenge)Image processing for high throughput sequencing.Wide area clouds (won SC 09 BWC with 100 Gbps wide area computation) New ensemble-based algorithms for treesGraph processing38
- 45. Cistrack Web Portal & WidgetsCistrack Elastic Cloud ServicesCistrack DatabaseAnalysis Pipelines & Re-analysis ServicesCistrack Large Data Cloud ServicesIngestion Services
- 46. Thank youFor more information, please see blog.rgrossman.com40