Data Structures and Performance for Scientific Computing with Hadoop and Dumbo (ICME MR 2012)
0 likes295 views
This document discusses matrix storage and data serialization techniques for scientific computing with Hadoop and Dumbo. It provides examples of storing matrices in HDFS using different approaches like storing each row separately, storing two rows per record, or flattening the matrix into a single list. It also discusses optimizing data serialization and switching programming languages. The document then presents an example of outputting many small matrices to disk and compares two MapReduce implementations for computing the Cholesky QR decomposition, identifying which approach is usually better and why.
![Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
Dense matrix storage
A =
0
11 12 13 14
21 22 23 24
31 32 33 34
41 42 42 44
BB@
1
CCA
In HDFS:
h1; [11; 12; 13; 14]i
h2; [21; 22; 23; 24]i
h3; [31; 32; 33; 34]i
h4; [41; 42; 43; 44]i](https://image.slidesharecdn.com/mapreduce-2012-140912163635-phpapp02/85/Data-Structures-and-Performance-for-Scientific-Computing-with-Hadoop-and-Dumbo-ICME-MR-2012-5-320.jpg)
![Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
Two rows per record
or we might use:
h1; [[11; 12; 13; 14]; [21; 22; 23; 24]]i
h3; [[31; 32; 33; 34]; [41; 42; 43; 44]]i](https://image.slidesharecdn.com/mapreduce-2012-140912163635-phpapp02/85/Data-Structures-and-Performance-for-Scientific-Computing-with-Hadoop-and-Dumbo-ICME-MR-2012-6-320.jpg)
![Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
Flattened list
or maybe
h1; [11; 12; 13; 14; 21; 22; 23; 24]i
h3; [31; 32; 33; 34; 41; 42; 43; 44]i
... but we do lose information here (maybe it's not important)](https://image.slidesharecdn.com/mapreduce-2012-140912163635-phpapp02/85/Data-Structures-and-Performance-for-Scientific-Computing-with-Hadoop-and-Dumbo-ICME-MR-2012-7-320.jpg)
![Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
Full matrix
or maybe
h1; [[11; 12; 13; 14]; [21; 22; 23; 24]; [31; 32; 33; 34]; [41; 42; 43; 44]]i](https://image.slidesharecdn.com/mapreduce-2012-140912163635-phpapp02/85/Data-Structures-and-Performance-for-Scientific-Computing-with-Hadoop-and-Dumbo-ICME-MR-2012-8-320.jpg)
Data Structures and Performance for Scientific Computing with Hadoop and Dumbo (ICME MR 2012)
- 1. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Data Structures and Performance for Scienti
- 2. c Computing with Hadoop and Dumbo Austin R. Benson Computer Sciences Division, UC-Berkeley ICME, Stanford University May 15, 2012
- 3. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR 1 1 Matrix storage 2 Data 3 Example: outputting many small matrices 4 Example: Cholesky QR
- 4. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Dense matrix storage A = 0 11 12 13 14 21 22 23 24 31 32 33 34 41 42 42 44 BB@ 1 CCA How do we store the matrix in HDFS?
- 5. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Dense matrix storage A = 0 11 12 13 14 21 22 23 24 31 32 33 34 41 42 42 44 BB@ 1 CCA In HDFS: h1; [11; 12; 13; 14]i h2; [21; 22; 23; 24]i h3; [31; 32; 33; 34]i h4; [41; 42; 43; 44]i
- 6. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Two rows per record or we might use: h1; [[11; 12; 13; 14]; [21; 22; 23; 24]]i h3; [[31; 32; 33; 34]; [41; 42; 43; 44]]i
- 7. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Flattened list or maybe h1; [11; 12; 13; 14; 21; 22; 23; 24]i h3; [31; 32; 33; 34; 41; 42; 43; 44]i ... but we do lose information here (maybe it's not important)
- 8. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Full matrix or maybe h1; [[11; 12; 13; 14]; [21; 22; 23; 24]; [31; 32; 33; 34]; [41; 42; 43; 44]]i
- 9. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR What is the "best" way?
- 10. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR What is the "best" way? Depends on the application... we will look at an example later.
- 11. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR 2 1 Matrix storage 2 Data 3 Example: outputting many small matrices 4 Example: Cholesky QR
- 12. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Data Serialization Small optimizations ! 2.5x speedup! *all data from the NERSC Magellan cluster
- 13. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Data Serialization Same experiment but dierent matrix size (200 columns): Again, 2.5x speedup!
- 14. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Languages Switching from Python to C++... same general trend
- 15. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR More speedups Algorithm performance isn't the only place where we see speedups
- 16. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Why can we expect these speedups? These are not high-performance implementations. We care about I/O performance.
- 17. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR 3 1 Matrix storage 2 Data 3 Example: outputting many small matrices 4 Example: Cholesky QR
- 18. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Suppose we need to write many small matrices to disk.
- 19. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Code Code: git clone git://github.com/icme/mapreduce-workshop.git cd mapreduce-workshop/arbenson Files: speed test.py (tester) small matrix test.py (driver)
- 20. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
- 21. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
- 22. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
- 23. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR
- 24. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR 4 1 Matrix storage 2 Data 3 Example: outputting many small matrices 4 Example: Cholesky QR
- 25. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Algorithm Cholesky QR: R = chol(ATA, 'upper')
- 26. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Implementation for MapReduce
- 27. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Mapper implementation Which of these implementations is better?
- 28. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Mapper implementation Which of these implementations is better? Answer: the one on the left (usually)
- 29. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Why? 1 Shue time 2 Reduce bottleneck However, the left implementation could run out of memory.
- 30. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Mapper implementation Can we do better? Yes
- 31. Matrix storage Data Example: outputting many small matrices Example: Cholesky QR Questions? Austin R. Benson arbenson@gmail.com https://github.com/arbenson/mrtsqr