.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure
0 likes486 views
The document discusses Apache Hadoop-based services for Windows Azure, including key components like Hadoop Distributed File System (HDFS), MapReduce, and tools such as Zookeeper and Pig. It emphasizes the differences between traditional RDBMS and MapReduce in terms of data handling, access types, and scalability. Additionally, it mentions practical implementations and demonstrations, highlighting the integration of Hadoop with various tools and remote access options.
![Map/Reduce
DataNode DataNode DataNode 0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1952,-11
1950,22
Map Map Map 1950,55
1950,33
Sort Sort Sort 1949,0
1950,[22,33,55]
Shuffle Shuffle Shuffle 1952,-11
Reduce
1949,0
1950,55
1952,-11](https://image.slidesharecdn.com/bigdataindercloud-dnugrna-120509030747-phpapp02/85/NET-Usergroup-Rhein-Neckar-Big-Data-in-der-Cloud-Apache-Hadoop-based-Services-fur-Windows-Azure-7-320.jpg)
![Map/Reduce
DataNode DataNode DataNode 0067011990999991950051507004+68750
0043011990999991950051512004+68750
0043011990999991950051518004+68750
0043012650999991949032412004+62300
0043012650999991949032418004+62300
1949,0
1952,-11
1950,22
Map Map Map 1950,55
1950,33
1949,0 1952,-11
Combine Combine Combine 1950,55 1950,33
Sort Sort Sort 1949,0
1950,[33,55]
Shuffle Shuffle Shuffle 1952,-11
Reduce
1949,0
1950,55
1952,-11](https://image.slidesharecdn.com/bigdataindercloud-dnugrna-120509030747-phpapp02/85/NET-Usergroup-Rhein-Neckar-Big-Data-in-der-Cloud-Apache-Hadoop-based-Services-fur-Windows-Azure-8-320.jpg)
.NET Usergroup Rhein-Neckar: Big Data in der Cloud - Apache Hadoop-based Services für Windows Azure
- 1. Apache Hadoop-based Services für Windows Azure Sascha Dittmann Software Developer / Solution Architect Twitter: @SaschaDittmann Blog: http://www.sascha-dittmann.de
- 2. Apache Hadoop & Co Zookeeper Pig
- 3. Hadoop Distributed File System Cluster Startvorgang
- 4. Hadoop Distributed File System Ausfall des Namenodes (Failover)
- 5. Hadoop Distributed File System Benuteranfrage ① ② ② ②
- 6. Hadoop Distributed File System Portable Operating System Interface (POSIX) Replikation auf mehrere Datenknoten js> #ls input/ncdc Found 9 items drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/_distcp_logs_g0dedn drwxr-xr-x - Sascha supergroup 0 2012-04-24 12:04 /user/Sascha/input/ncdc/_distcp_logs_ofj0u6 drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:09 /user/Sascha/input/ncdc/all drwxr-xr-x - Sascha supergroup 0 2012-04-24 13:01 /user/Sascha/input/ncdc/all2 drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/metadata drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro drwxr-xr-x - Sascha supergroup 0 2012-04-23 13:06 /user/Sascha/input/ncdc/micro-tab -rw-r--r-- 3 Sascha supergroup 529 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt -rw-r--r-- 3 Sascha supergroup 168 2012-04-23 13:06 /user/Sascha/input/ncdc/sample.txt.gz
- 7. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[22,33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11
- 8. Map/Reduce DataNode DataNode DataNode 0067011990999991950051507004+68750 0043011990999991950051512004+68750 0043011990999991950051518004+68750 0043012650999991949032412004+62300 0043012650999991949032418004+62300 1949,0 1952,-11 1950,22 Map Map Map 1950,55 1950,33 1949,0 1952,-11 Combine Combine Combine 1950,55 1950,33 Sort Sort Sort 1949,0 1950,[33,55] Shuffle Shuffle Shuffle 1952,-11 Reduce 1949,0 1950,55 1952,-11
- 9. RDBMS vs. Map/Reduce RDBMS Map/Reduce Datenmenge Gigabytes Petabytes Zugriff Interaktiv und Batch Batch Lese- / Schreibzugriffe Viele Lese- und Einmaliges Schreiben Schreibzugriffe Viele Lesezugriffe Datenstruktur Statisches Schema Dynamisches Schema Datenintegrität Hoch Niedrig Skalierverhalten Nicht-Linear Linear
- 10. Apache Hadoop & Co Zookeeper Pig
- 11. Demos Hadoop Dashboard Interactive Console Remote Desktop Nutzung des WA Storage Map/Reduce via JavaScript C# Streaming Power Pivot
- 12. Cloud Bloggers Die Blogs der deutschen Cloud Computing-Community Link: http://cloudbloggers.de