Hw09 Hadoop Db
3 likes1,507 views
HadoopDB is a hybrid system developed at Yale University that combines technologies from MapReduce and parallel database systems. It uses Hadoop to coordinate multiple independent PostgreSQL databases to analyze massive datasets across thousands of nodes. Evaluation results showed that HadoopDB approaches the performance of parallel databases for structured data workloads, and performs comparably to Hadoop for unstructured data, while also scaling as well as Hadoop. HadoopDB is open source and provides an alternative to commercial parallel databases for large-scale data analysis.
Hw09 Hadoop Db
- 1. HadoopDB: An open source hybrid of MapReduce and DBMS technologies Azza Abouzeid, Kamil Bajda-Pawlikowski Daniel J. Abadi, Avi Silberschatz Yale University http://hadoopdb.sourceforge.net October 2, 2009 HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads. Azza Abouzeid, Kamil Bajda-Pawlikowski, Daniel J. Abadi, Avi Silberschatz, Alex Rasin. In Proceedings of VLDB, 2009.
- 2. Introduction Candidates Differences HadoopDB Evaluation Conclusion Motivation Major Trends 1 Data explosion: Automation of business processes, proliferation of digital devices. eBay has a 6.5 PB warehouse, Yahoo! Everest has 10 PB. 2 Analysis over raw data Yale University, HadoopWorld 2009 HadoopDB 2/24
- 3. Introduction Candidates Differences HadoopDB Evaluation Conclusion Motivation Major Trends 1 Data explosion: Automation of business processes, proliferation of digital devices. eBay has a 6.5 PB warehouse, Yahoo! Everest has 10 PB. 2 Analysis over raw data Bottom line Analyzing massive structured data on 1000s of shared-nothing nodes. Yale University, HadoopWorld 2009 HadoopDB 2/24
- 4. Introduction Candidates Differences HadoopDB Evaluation Conclusion Motivation Sales Record Example Consider a large data set of sales log records, each consisting of sales information including: 1 a date of sale 2 a price We would like to take the log records and generate a report showing the total sales for each year. Question: How do we generate this report efficiently and cheaply over massive data contained in a shared-nothing cluster of 1000s of machines? Yale University, HadoopWorld 2009 HadoopDB 3/24
- 5. Introduction Candidates Differences HadoopDB Evaluation Conclusion MapReduce Parallel Databases MapReduce (Hadoop) MapReduce is a programming model which specifies: A map function that processes a key/value pair to generate a set of intermediate key/value pairs, A reduce function that merges all intermediate values associated with the same intermediate key. Hadoop is a MapReduce implementation for processing large data sets over 1000s of nodes. Maps (and Reduces) run independently of each other over blocks of data distributed across a cluster. Yale University, HadoopWorld 2009 HadoopDB 4/24
- 6. Introduction Candidates Differences HadoopDB Evaluation Conclusion MapReduce Parallel Databases Sales Record Example using Hadoop Query: Calculate total sales for each year. We write a MapReduce program: Map: Takes log records and extracts a key-value pair of year and sale price in dollars. Outputs the key-value pairs. Shuffle: Hadoop automatically partitions the key-value pairs by year to the nodes executing the Reduce function Reduce: Simply sums up all the dollar values for a year. Yale University, HadoopWorld 2009 HadoopDB 5/24
- 7. Introduction Candidates Differences HadoopDB Evaluation Conclusion MapReduce Parallel Databases Relational Databases Suppose that the data is stored in a relational database system, the sales record example could be expressed in SQL as: SELECT YEAR(date) AS year, SUM(price) FROM sales GROUP BY year The execution plan is: projection(year,price) → hash aggregation(year,price) . Question: How do we process this efficiently if the data is very large? Yale University, HadoopWorld 2009 HadoopDB 6/24
- 8. Introduction Candidates Differences HadoopDB Evaluation Conclusion MapReduce Parallel Databases Parallel Databases Parallel Databases are like single-node databases except: Data is partitioned across nodes Individual relational operations can be executed in parallel xxx SELECT YEAR(date) AS year, SUM(price) FROM sales GROUP BY year Execution plan for the query: projection(year,price) → partial hash aggregation(year,price) → partitioning(year) → final aggregation(year,price) . Note that the execution plan resembles the map and reduce phases of Hadoop. Yale University, HadoopWorld 2009 HadoopDB 7/24
- 9. Introduction Candidates Differences HadoopDB Evaluation Conclusion Differences between Parallel Databases and Hadoop Yale University, HadoopWorld 2009 HadoopDB 8/24
- 10. Introduction Candidates Differences HadoopDB Evaluation Conclusion Differences between Parallel Databases and Hadoop Yale University, HadoopWorld 2009 HadoopDB 8/24
- 11. Introduction Candidates Differences HadoopDB Evaluation Conclusion To summarize Yale University, HadoopWorld 2009 HadoopDB 9/24
- 12. At Yale, we looked beyond the differences ...
- 13. At Yale, we looked beyond the differences ...
- 14. and we discovered ... Basic design idea Multiple, independent, single node databases coordinated by ... that they complete each other Hadoop. http://i214.photobucket.com/albums/cc19/brittanybutton/elephants.jpg
- 15. Introduction Candidates Differences HadoopDB Evaluation Conclusion Background Architecture SMS Hadoop Basics Yale University, HadoopWorld 2009 HadoopDB 12/24
- 16. Introduction Candidates Differences HadoopDB Evaluation Conclusion Background Architecture SMS Architecture Yale University, HadoopWorld 2009 HadoopDB 13/24
- 17. Introduction Candidates Differences HadoopDB Evaluation Conclusion Background Architecture SMS SQL-MR-SQL SELECT YEAR(saleDate), SUM(revenue) FROM sales GROUP BY YEAR(saleDate); Yale University, HadoopWorld 2009 HadoopDB 14/24
- 18. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Evaluating HadoopDB Compare HadoopDB to 1 Hadoop 2 Parallel databases (Vertica, DBMS-X) Features: 1 Performance: We expected HadoopDB to approach the performance of parallel databases 2 Scalability: We expected HadoopDB to scale as well as Hadoop We ran the Pavlo et al. SIGMOD’09 benchmark on Amazon EC2 clusters of 10, 50, 100 nodes. Yale University, HadoopWorld 2009 HadoopDB 15/24
- 19. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Load 1600 Vertica DB-X HadoopDB Hadoop Vertica DB-X Thousands 1400 HadoopDB Hadoop 50 1200 40 1000 seconds seconds 800 30 600 20 400 164 161 141 139 200 10 100 92 77 47 43 0 10 nodes 50 nodes 100 nodes 0 10 nodes 50 nodes 100 nodes Random Unstructured Data Structured data (20GB/node) (535MB/node) Yale University, HadoopWorld 2009 HadoopDB 16/24
- 20. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Performance: Grep Task 70 Vertica DB-X HadoopDB Hadoop 60 1 Full table scan, highly 50 selective filter 40 2 Random data, no seconds 30 room for indexing 20 3 Hadoop overhead outweighs query 10 processing time in 0 10 nodes 50 nodes 100 nodes single-node databases SELECT * FROM grep WHERE field LIKE ‘%xyz%’; Yale University, HadoopWorld 2009 HadoopDB 17/24
- 21. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Performance: Join Task 2000 Vertica DB-X HadoopDB Hadoop 1800 1600 1400 1200 seconds 1000 800 1 No full table scan due 600 to clustered indexing 300.5 224.2 400 2 Hash partitioning and 126.4 67.7 200 34.7 31.9 29.4 28.0 20.6 0 efficient join 10 nodes 50 nodes 100 nodes algorithm SELECT sourceIP, AVG(pageRank), SUM(adRevenue) FROM rankings, uservisits WHERE pageURL=destURL AND visitDate BETWEEN 2000-1-15 AND 2000-1-22 GROUP BY sourceIP ORDER BY SUM(adRevenue) DESC LIMIT 1; Yale University, HadoopWorld 2009 HadoopDB 18/24
- 22. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Performance: Bottom Line 1 Unstructured data HadoopDB’s performance matches Hadoop 2 Structured data HadoopDB’s performance is close to parallel databases Yale University, HadoopWorld 2009 HadoopDB 19/24
- 23. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Scalability: Setup 1 Simple aggregation task - full table scan 2 Data replicated across 10 nodes 3 Fault-tolerance: Kill a node halfway 4 Fluctuation-tolerance: Slow down a node for the entire experiment Yale University, HadoopWorld 2009 HadoopDB 20/24
- 24. Introduction Candidates Differences HadoopDB Evaluation Conclusion Hypotheses Load Performance Scalability Scalability: Results 200% Vertica 1 HadoopDB and HadoopDB 180% Hadoop Hadoop take 160% advantage of runtime percentage slowdown 140% scheduling by 120% 100% splitting data into 80% chunks or blocks 60% 2 Parallel databases 40% restart entire query on 20% 0% node failure or wait Fault-tolerance Fluctuation-tolerance for the slowest node Yale University, HadoopWorld 2009 HadoopDB 21/24
- 25. Introduction Candidates Differences HadoopDB Evaluation Conclusion Summary Future To summarize HadoopDB ... 1 is a hybrid of DBMS and MapReduce 2 scales better than commercial parallel databases 3 is as fault-tolerant as Hadoop 4 approaches the performance of parallel databases 5 is free and open-source http://hadoopdb.sourceforge.net Yale University, HadoopWorld 2009 HadoopDB 22/24
- 26. Introduction Candidates Differences HadoopDB Evaluation Conclusion Summary Future Future work Engineering work: 1 Full SQL support in SMS 2 Data compression 3 Integration with other open source databases 4 Full automation of the loading and replication process 5 Out-of-the box deployment 6 We’re hiring! Research work: Incremental loading and on-the-fly repartitioning Dynamically adjusting fault-tolerance levels based on failure rate Yale University, HadoopWorld 2009 HadoopDB 23/24
- 27. Thank You ... We welcome all thoughts on how to raise HadoopDB ... http://www.jpbutler.com/thailand/images/elephant-8-days-old.jpg