Sector - Presentation at Cloud Computing & Its Applications 2009
Download as PPTX, PDF2 likes533 views
Sector is an open-source cloud platform for data-intensive computing, designed for reliability and security, including HIPAA compliance. The platform supports user-defined functions (UDFs), MapReduce, and Python, running on commodity computer racks. Performance benchmarks show Sector outperforms Hadoop in various tests, utilizing a specialized network transport protocol (UDT) for high-speed data transfer in wide-area cloud environments.
Sector - Presentation at Cloud Computing & Its Applications 2009
- 1. Sector: An Open Source Cloud for Data Intensive ComputingRobert GrossmanUniversity of Illinois at ChicagoOpen Data GroupOctober 20, 2009
- 2. Part 1. Sector2http://sector.sourceforge.net
- 3. Sector OverviewSector is fastest open source large data cloudAs measured by MalStone & TerasortSector is easy to programSupports UDFs, MapReduce & Python over streamsSector is secureA HIPAA compliant Sector cloud is being set upSector is reliableSector v1.24 has a backup master node server3
- 4. About Sector YunhongGu from the Laboratory for Advanced Computing at the University of Illinois at Chicago is the Lead Developer of Sector.Sector is open source (BSD License) and available from sector.sourceforge.netThe current version is 1.24a4
- 5. Target ConfigurationsSector is designed to run on racks of commodity computersTypical rack configuration today (Oct, 2009)Rack of 32 quad-core 1U computersEach computer has 4 x 1TB disksEach computer has 1 Gbps connection to a top of a rack switchSometimes these are called Raywulf clusters5
- 6. Google’s Large Data CloudCompute ServicesData ServicesStorage Services6ApplicationsGoogle’s MapReduceGoogle’s BigTableGoogle File System (GFS)Google’s Stack
- 7. Hadoop’s Large Data CloudCompute ServicesStorage Services7ApplicationsHadoop’sMapReduceData ServicesHadoop Distributed File System (HDFS)Hadoop’s Stack
- 8. Sector’s Large Data Cloud8ApplicationsCompute ServicesSphere’s UDFsData ServicesSector’s Distributed File System (SDFS)Storage ServicesUDP-based Data Transport Protocol (UDT)Routing & Transport ServicesSector’s Stack
- 9. Comparing Sector and Hadoop9
- 10. Terasort - Sector vsHadoop PerformanceSector/Sphere 1.24a, Hadoop 0.20.1 with no replication on Phase 2 of Open Cloud Testbed with co-located racks.
- 11. MalStone (OCC-Developed Benchmark)Sector/Sphere 1.20, Hadoop 0.18.3 with no replication on Phase 1 of Open Cloud Testbed in a single rack. Data consisted of 20 nodes with 500 million 100-byte records / node.
- 12. How Do You Program A Data Center?12
- 13. Idea 1 – Support UDF’s Over Data CenterThink of MapReduce asMap acting on (text) recordsWith fixed Shuffle and SortFollowed by Reducing acting on (text) recordsWe generalize this framework as follows:Support a sequence of User Defined Functions (UDF) acting on segments (=chunks) of files.MapReduce is one special case consisting of a user defined Map, a system-defined shuffle and sort, and a user defined reduceIn both cases, framework takes care of assigning nodes to process data, restarting failed processes, etc.13
- 14. Applying UDF using Sector/Sphere141. Split dataApplicationSphere ClientInput streamSPESPESPE2. Locate & schedule Sphere Processing Engine (SPE)3. Collect resultsOutputstream
- 15. Sector Programming ModelSector dataset consists of one or more physical filesSphere applies User Defined Functions over streams of data consisting of data segmentsData segments can be data records, collections of data records, or filesExample of UDFs: Map function, Reduce function, Split function for CART, etc.Outputs of UDFs can be returned to originating node, written to local node, or shuffled to another node.15
- 16. How Do Move Data in a Cloud & Between Clouds?16Option 1: Use TCP and close your eyes.Option 2: ?????
- 17. Idea 2: Sector is Built on Top of UDT17UDT is a specialized network transport protocol.
- 18. UDT can take advantage of wide area high performance 10 Gbps network
- 19. Sector is a wide area distributed file system built over UDT.
- 20. Sector is layered over the native file system (vs being a block-based file system).UDT Has Been Downloaded 25,000+ Times18Sterling Commerceudt.sourceforge.netMovie2MeGlobusPower FolderNifty TVhttp://udt.sourceforge.net
- 21. (x)UDTScalable TCPHighSpeed TCPAIMD (TCP NewReno)xAlternatives to TCP – Decreasing Increases AIMD Protocolsincrease of packet sending rate xdecrease factor
- 22. UDT Makes Wide Area Clouds PossibleUsing UDT, Sector can take advantage of wide area high performance networks (10+ Gbps)2010 Gbps per application
- 24. Idea 3: Add Security From the StartSecurity ServerSecurity server maintains information about users and slaves.User access control: password and client IP address.File level access control.Messages are encrypted over SSL. Certificate is used for authentication.Sector is HIPAA capable.MasterClientSSLSSLAAAdataSlaves
- 25. For More Information About SectorYunhongGu and Robert L Grossman, Sector and Sphere: Towards Simplified Storage and Processing of Large Scale Distributed Data, Philosophical Transactions of the Royal Society A, Volume 367, Number 1897, pages 2429--2445, 2009http://arxiv.org/abs/0809.1181http://rsta.royalsocietypublishing.org/content/367/1897/242923
- 26. For Related InformationRelated information can be found at:blog.rgrossman.comwww.rgrossman.com24
- 27. Sector Sponsors