Scalable Parallel Performance Measurement with the Scalasca Toolset
0 likes722 views
This document discusses the Scalasca toolset for scalable parallel performance measurement. It summarizes that Scalasca can instrument large parallel applications to collect event traces and analyze them to find inefficiencies. Specifically, it can identify patterns like late senders/receivers and indirect waiting. The document provides examples analyzing a CESM sea ice model to identify direct and indirect waiting caused by load imbalances. It also discusses supporting Intel MIC architectures and acknowledging the Scalasca team and sponsors.
Scalable Parallel Performance Measurement with the Scalasca Toolset
- 1. MitgliedderHelmholtz-Gemeinschaft Scalable Parallel Performance Measurement with the Scalasca Toolset Bernd Mohr June 2013
- 2. June 2013 JSC 2 Parallel Architectures: State of the Art Network or Switch ... N0 N1 Nk Inter- connect P0 Pn ... Memory A0 Am ... Inter- connect P0 Pn ... Memory A0 Am ... Inter- connect P0 Pn ... A0 Am ... Memory Pi Core0 Core1 Corer L10 L11 L1 L20 L2r/2 L30 ... ... Aj Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router Router or SMP NUMA
- 3. June 2013 JSC 3 Parallel Performance Challenges • Current and future systems (will) consist of Complex configurations With a huge number of components Very likely heterogeneous • Deep software hierarchies of large, complex software components will be required to make use of such systems Sophisticated integrated performance measurement, analysis, and optimization capabilities will be required to efficiently operate such systems Tools which provide insight not just numbers or charts needed!
- 4. June 2013 JSC 4 “A picture is worth 1000 words…” • “Real world” example• MPI ring program
- 5. June 2013 JSC 5 “What about 1000’s of pictures?” (with 100’s of menu options)
- 6. June 2013 JSC 6 Example Automatic Analysis: Late Sender
- 7. June 2013 JSC 7 Scalasca: Example MPI Patterns time process ENTER EXIT SEND RECV COLLEXIT (a) Late Sender time process (b) Late Receiver time process (d) Wait at N x N time process (c) Late Sender / Wrong Order
- 8. June 2013 JSC 8 The Scalasca Project • Scalable Analysis of Large Scale Applications • Approach Instrument C, C++, and Fortran parallel applications Based on MPI, OpenMP, SHMEM, or hybrid Option 1: scalable call-path profiling Option 2: scalable event trace analysis Collect event traces Search trace for event patterns representing inefficiencies Categorize and rank inefficiencies found • Supports MPI 2.2 (P2P, collectives, RMA, IO) and OpenMP 3.0 (excl. nesting) http://www.scalasca.org/
- 9. June 2013 JSC 9 Scalasca Example: CESM Sea Ice Module Late Sender Analysis • Finds waiting at MPI_Waitall() inside ice boundary halo update • Shows distribution of imbalance across system and ranks
- 10. June 2013 JSC 10 Scalasca Example: CESM Sea Ice Module Late Sender Analysis + Application Topology • Shows distribution of imbalance over topology • MPI topologies are automatically captured
- 11. June 2013 JSC 11time Scalasca Root Cause Analysis • Root-cause analysis Wait states typically caused by load or communication imbalances earlier in the program Waiting time can also propagate (e.g., indirect waiting time) Enhanced performance analysis to find the root cause of wait states • Approach Distinguish between direct and indirect waiting time Identify call path/process combinations delaying other processes and causing first order waiting time Identify original delay Recv Send Send foo foo foo bar bar Recv A B C cause Recv Recv Direct waitIndirect wait Recv barDELAY
- 12. June 2013 JSC 12 Scalasca Example: CESM Sea Ice Module Direct Wait Time Analysis • Direct wait caused by ranks processing areas near the north and south ice borders
- 13. June 2013 JSC 13 Scalasca Example: CESM Sea Ice Module Indirect Wait Time Analysis • Indirect waits occurs for ranks processing warmer areas
- 14. June 2013 JSC 14 Scalasca Example: CESM Sea Ice Module Delay Costs Analysis • Delays NOT caused on ranks processing ice!
- 15. June 2013 JSC 15 NEW: Scalasca on Intel MIC Example: • TACC Stampede • NAS BT-MZ code • MPI/OpenMP • 8x16 CPU threads (2 MPI/node) • 60x16 MIC threads (15 MPI/MIC) Supported modes • Host-only or MIC-only • Symmetric Not yet supported modes • Offload
- 16. June 2013 JSC 16 Acknowledgements • Scalasca team (JSC) (GRS) • Sponsors Michael Knobloch Bernd Mohr Peter Philippen Markus Geimer Daniel Lorenz Christian Rössel David Böhme Marc-André Hermanns Pavel Saviankou Marc Schlütter Ilja Zhukov Alexandre Strube Brian Wylie Felix Wolf Anke Visser Monika Lücke Aamer Shah Alexandru Calotoiu Jie Jiang Sergei Shudler Guoyong Mao Philipp Gschwandtner
- 17. June 2013 JSC 17 Questions? • Check out http://www.scalasca.org • Or contact us at scalasca@fz-juelich.de