Map Reduce in the Clouds (http://salsahpc.indiana.edu/mapreduceroles4azure/)

Editor's Notes

• #3: The utility computing model introduced by cloud computing combined with the rich set of cloud infrastructure services offers a very viable alternative to traditional servers and computing clusters. MapReduce distributed data processing architecture has become the weapon of choice for data-intensive analyses in the clouds and in commodity clusters due to its excellent fault tolerance features, scalability and the ease of use. Currently, there are several options for using MapReduce in cloud environments, such as using MapReduce as a service, setting up one’s own MapReduce cluster on cloud instances, or using specialized cloud MapReduce runtimes that take advantage of cloud infrastructure services. In this paper, we introduce AzureMapReduce, a novel MapReduce runtime built using the Microsoft Azure cloud infrastructure services. AzureMapReduce architecture successfully leverages the high latency, eventually consistent, yet highly scalable Azure infrastructure services to provide an efficient, on demand alternative to traditional MapReduce clusters. Further we evaluate the use and performance of MapReduce frameworks, including AzureMapReduce, in cloud environments for scientific applications using sequence assembly and sequence alignment as use cases.
• #6: Data storage: Clouds typically provide a variety of storage options, such as off-instance cloud storage (e.g.: Amazon S3), mountable off-instance block storage (e.g.: Amazon EBS) as well as virtualized instance storage (persistent for the lifetime of the instance), which can be used to set up a file system similar to HDFS [13]. The choice of the storage best-suited to the particular MapReduce deployment plays a crucial role as the performance of data intensive applications rely a lot on the storage location and on the storage bandwidth.Metadata storage: MapReduce frameworks need to maintain metadata information to manage the jobs as well as the infrastructure. This metadata needs to be stored reliability ensuring good scalability and the accessibility to avoid single point of failures and performance bottlenecks to the MapReduce computation.Communication consistency and scalability: Cloud infrastructures are known to exhibit inter-node I/O performance fluctuations (due to shared network, unknown topology), which affect the intermediate data transfer performance of MapReduce applications.Performance consistency (sustained performance): Clouds are implemented as shared infrastructures operating using virtual machines. It’s possible for the performance to fluctuate based the load of the underlying infrastructure services as well as based on the load from other users on the shared physical node which hosts the virtual machine (see Section VII).Reliability (Node failures): Node failures are to be expected whenever large numbers of nodes are utilized for computations. But they become more prevalent when virtual instances are running on top of non-dedicated hardware. While MapReduce frameworks can recover jobs from worker node failures, master node (nodes which store meta-data, which handle job scheduling queue, etc) failures can become disastrous.Choosing a suitable instance type: Clouds offer users several types of instance options, with different configurations and price points (See Sections B and D). It’s important to select the best matching instance type, both in terms of performance as well as monetary wise, for a particular MapReduce job.Logging: Cloud instance storage is preserved only for the lifetime of the instance. Hence, information logged to the instance storage would be lost after the instance termination. This can be crucial if one needs to process the logs afterwards, for an example to identify a software-caused instance failure. On the other hand, performing excessive logging to a bandwidth limited off-instance storage location can become a performance bottleneck for the MapReduce computation.
• #10: Client driver loads the map & reduce tasks to queues in parallel using TPL..Create the task monitoring table. Standalone client or a web client. Can wait for completion.Explain the advantages of using Azure queues.Explain the advantages of using Azure table.. Scalability. Ease of use.. No maintenance overhead. No need to install DB. Easily visualize using a webrole.
• #11: Map & Reduce workers pick up map tasks from the queue
• #12: Map workers download data from Blob storage and start processing- – update the status in the task monitoring table.Advantages of blob storage.Custom input/output formats & keys..
• #13: Finished Map tasks upload result data sets to Azure Storage and then add entries for the respective reduce task tables. – update the status. Get the next task from the queue and start processing it.Custom part
• #14: Reduce tasks notice the intermediate data product meta-data in reduce task tables and start downloading them -> update the reduce task tablesThis happens when the map tasks are actually processing the next set of map tasks..
• #15: Reduce tasks start reducing, when all the map tasks are finished and when the respective reduce tasks are finish downloading the intermediate data products.Custom output formats
• #16: Global barrier…
• #17: Idataplex - Two quad-core CPUs (Intel Xeon CPU E5410 2.33GHz) 16 GB memory, Gigabit Ethernet network interface
• #18: Use block decompositionLower triangle only using load balancing algorithmEach row block is collected by reducers.Relatively small amount of input data, but large intermediate and output data.
• #19: ~123 million sequence alignments, for under 30$ with zero up front hardware cost,
• #22: SWG - In these tests, 32 cores were used to align 4000 sequences. Standard deviations of 1.56% for EMR and 2.25% for AzureMapReduce.