SlideShare a Scribd company logo

HDR Defence - Software Abstractions for Parallel Architectures

Joel Falcou, profile picture
Joel Falcou

The document discusses advancements in software abstractions for parallel architectures, particularly in scientific computing, emphasizing the transition from traditional experimentation to simulation-based research. It highlights the challenges posed by increasing hardware complexity and the need for efficient, expressive parallel programming models, such as bulk synchronous parallelism and parallel skeletons. The strategies proposed include designing domain-specific embedded languages and employing generative programming to enhance performance while maintaining ease of use across various architectures.

Software
1 of 55
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
Software Abstractions for Parallel Architectures Joel Falcou LRI - CNRS - INRIA HDR Thesis Defense 12/01/2014
The Paradigm Change in Science From Experiments to Simulations Simulations is now an integral part of the Scientic Method Scientic Computing enables larger, faster, more accurate Research Fast Simulation is Time Travel as scientic results are now more readily available Local Galaxy Cluster Simulation - Illustris project Computing is rst and foremost a mainstream science tool 2 of 41
The Paradigm Change in Science The Parallel Hell Heat Wall: Growing cores instead of GHz Hierarchical and heterogeneous parallel systems are the norm The Free Lunch is over as hardware complexity rises faster than the average developer skills Local Galaxy Cluster Simulation - Illustris project The real challenge in HPC is the Expressiveness/Efficiency War 2 of 41
The Expressiveness/Efficiency War Single Core Era Performance Expressiveness C/Fort. C++ Java Multi-Core/SIMD Era Performance Expressiveness Sequential Threads SIMD Heterogenous Era Performance Expressiveness Sequential SIMD Threads GPU Phi Distributed As parallel systems complexity grows, the expressiveness gap turns into an ocean 3 of 41
Designing tools for Scientic Computing Objectives 1. Be non-disruptive 2. Domain driven optimizations 3. Provide intuitive API for the user 4. Support a wide architectural landscape 5. Be efficient 4 of 41
Designing tools for Scientic Computing Objectives 1. Be non-disruptive 2. Domain driven optimizations 3. Provide intuitive API for the user 4. Support a wide architectural landscape 5. Be efficient Our Approach Design tools as C++ libraries (1) Design these libraries as Domain Specic Embedded Languages (DSEL) (2+3) Use Parallel Programming Abstractions as parallel components (4) Use Generative Programming to deliver performance (5) 4 of 41
Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 5 of 41
Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap 6 of 41
Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap Available Models Performance centric: P-RAM, LOG-P, BSP Pattern centric: Futures, Skeletons Data centric: HTA, PGAS 6 of 41
Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap Available Models Performance centric: P-RAM, LOG-P, BSP Pattern centric: Futures, Skeletons Data centric: HTA, PGAS 6 of 41
Bulk Synchronous Parallelism [Valiant, McColl 90] Principles Machine Model Execution Model Analytic Cost Model C o m p u t e B a r r i e r C o m m Wmax h.g P0 P1 P2 P3 Superstep T Superstep T+1 LWmax h.g BSP Execution Model 7 of 41
Bulk Synchronous Parallelism [Valiant, McColl 90] Advantages Simple set of primitives Implementable on any kind of hardware Possibility to reason about BSP programs C o m p u t e B a r r i e r C o m m Wmax h.g P0 P1 P2 P3 Superstep T Superstep T+1 LWmax h.g BSP Execution Model 7 of 41
Parallel Skeletons [Cole 89] Principles There are patterns in parallel applications Those patterns can be generalized in Skeletons Applications are assembled as a combination of such patterns Functional point of view Skeletons are Higher-Order Functions Skeletons support a compositionnal semantic Applications become composition of state-less functions 8 of 41
Parallel Skeletons [Cole 89] Principles There are patterns in parallel applications Those patterns can be generalized in Skeletons Applications are assembled as a combination of such patterns Classical Skeletons Data parallel: map, fold, scan Task parallel: par, pipe, farm More complex: Distribuable Homomorphism, Divide & Conquer, … 8 of 41
Relevance to our Objectives Why using Parallel Skeletons ? Write code independant of parallel programming minutiae Composability supports hierarchical architectures Code is scalable and easy to maintain Why using BSP ? Cost model guide development Few primitives mean that intellectual burden is low Good medium for developping skeletons How to ensure performance of those models’ implementations ? 9 of 41
Domain Specic Embedded Languages Domain Specic Languages Non-Turing complete declarative languages Solve a single type of problems Express what to do instead of how to do it E.g: SQL, M, M, … From DSL to DSEL [Abrahams 2004] A DSL incorporates domain-specic notation, constructs, and abstractions as fundamental design considerations. A Domain Specic Embedded Languages (DSEL) is simply a library that meets the same criteria Generative Programming is one way to design such libraries 10 of 41
Generative Programming [Eisenecker 97] Domain Specific Application Description Generative Component Concrete Application Translator Parametric Sub-components 11 of 41
Meta-programming as a Tool Denition Meta-programming is the writing of computer programs that analyse, transform and generate other programs (or themselves) as their data. Meta-programmable Languages metaOCAML : runtime code generation via code quoting Template Haskell : compile-time code generation via templates C++ : compile-time code generation via templates C++ meta-programming Relies on the Turing-complete C++  sub-language Handles types and integral constants at compile-time  classes and functions act as code quoting 12 of 41
The Expression Templates Idiom Principles Relies on extensive operator overloading Carries semantic information around code fragment Introduces DSLs without disrupting dev. chain matrix x(h,w),a(h,w),b(h,w); x = cos(a) + (b*a); expr<assign ,expr<matrix&> ,expr<plus , expr<cos ,expr<matrix&> > , expr<multiplies ,expr<matrix&> ,expr<matrix&> > >(x,a,b); + *cos a ab = x #pragma omp parallel for for(int j=0;j<h;++j) { for(int i=0;i<w;++i) { x(j,i) = cos(a(j,i)) + ( b(j,i) * a(j,i) ); } } Arbitrary Transforms applied on the meta-AST General Principles of Expression Templates 13 of 41
The Expression Templates Idiom Advantages Generic implementation becomes self-aware of optimizations API abstraction level is arbitrary high Accessible through high-level tools like B.P matrix x(h,w),a(h,w),b(h,w); x = cos(a) + (b*a); expr<assign ,expr<matrix&> ,expr<plus , expr<cos ,expr<matrix&> > , expr<multiplies ,expr<matrix&> ,expr<matrix&> > >(x,a,b); + *cos a ab = x #pragma omp parallel for for(int j=0;j<h;++j) { for(int i=0;i<w;++i) { x(j,i) = cos(a(j,i)) + ( b(j,i) * a(j,i) ); } } Arbitrary Transforms applied on the meta-AST General Principles of Expression Templates 13 of 41
Our Contributions Our Strategy Applies DSEL generation techniques to parallel programming Maintains low cost of abstractions through meta-programming Maintains abstraction level via modern library design Our contributions Tools Pub. Scope Applications Quaff ParCo’06 MPI Skeletons Real-time 3D reconstruction SkellBE PACT’08 Skeleton on Cell BE Real-time Image processing BSP++ IJPP’12 MPI/OpenMP BSP Bioinformatics, Model Checking NT2 JPDC’14 Data Parallel Matlab Fluid Dynamics, Vision 14 of 41
Example of BSP++ Application Khaled Hamidouche PHD 2008-2011 in collab. with Univ. Brasilia BSP Smith & Waterman S&W computes DNA sequences alignment BSP++ implementation was written once and run on 7 different hardwares Efficiency of 95+% even on 6000 cores super-computer Platform MaxSize # Elements Speedup GCUPs cluster (MPI) 1,072,950 128 cores 73x 6.53 cluster (MPI/OpenMP) 1,072,950 128 cores 116x 10.41 OpenMP 1,072,950 16 cores 16x 0.40 CellBE 85,603 8 SPEs — 0.14 cluster of CellBEs 85,603 24 SPEs (8:24) 2.8x 0.37 Hopper(MPI) 5,303,436 3072 cores 260x 3.09 Hopper(MPI+OpenMP) 24,894,269 6144 cores 5664x 15,5 15 of 41
Second Look at our Contributions Development Limitations DSELs are mostly tied to the domain model Architecture support is often an afterthought Extensibility is difficult as many refactoring are required per architecture Example : No proper way to support GPUs with those implementation techniques 16 of 41
Second Look at our Contributions Development Limitations DSELs are mostly tied to the domain model Architecture support is often an afterthought Extensibility is difficult as many refactoring are required per architecture Example : No proper way to support GPUs with those implementation techniques Proposed Method Extends Generative Programming to take this architecture into account Provides an architecture description DSEL Integrates this description in the code generation process 16 of 41
Architecture Aware Generative Programming 17 of 41
Software refactoring Tools Issues Changes Quaff Raw skeletons API Re-engineered as part of NT2 SkellBE Too architecture specic Re-engineered as part of NT2 BSP++ Integration issues Integrate hybrid code generation NT2 Not easily extendable Integrate Quaff Skeleton models Boost.SIMD - Side product of NT2 restructuration Conclusion Skeletons are ne as parallel middleware Model based abstractions are not high level enough For low level architectures, the simplest model is often the best 18 of 41
Boost.SIMD Pierre Estérie PHD 2010-2014 Principles Provides simple C++ API over SIMD extensions Supports every Intel, PPC and ARM instructions sets Fully integrates with modern C++ idioms Sparse Tridiagonal Solver - collaboration with M. Baboulin and Y. wang 19 of 41
Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 20 of 41
The Numerical Template Toolbox Pierre Estérie PHD 2010-2014 NT2 as a Scientic Computing Library Provides a simple, M-like interface for users Provides high-performance computing entities and primitives Is easily extendable Components Uses Boost.SIMD for in-core optimizations Uses recursive parallel skeletons Supports task parallelism through Futures 21 of 41
The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M 22 of 41
The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le 22 of 41
The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le Add #include <nt2/nt2.hpp> and do cosmetic changes 22 of 41
The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le Add #include <nt2/nt2.hpp> and do cosmetic changes Compile the le and link with libnt2.a 22 of 41
NT2 - From M to C++ M code A1 = 1:1000; A2 = A1 + randn(size(A1)); X = lu(A1*A1’); rms = sqrt( sum(sqr(A1(:) - A2(:))) / numel(A1) ); NT2 code table <double > A1 = _(1. ,1000.); table <double > A2 = A1 + randn(size(A1)); table <double > X = lu( mtimes(A1, trans(A1) ); double rms = sqrt( sum(sqr(A1(_) - A2(_))) / numel(A1) ); 23 of 41
Parallel Skeletons extraction process A = B / sum(C+D); = A / B sum + C D fold transform 24 of 41
Parallel Skeletons extraction process A = B / sum(C+D); ; ; = A / B sum + C D fold transform = tmp sum + C D fold ⇒ = A / B tmp transform 25 of 41
From data to task parallelism Antoine Tran Tan PHD, 2012-2015 Limits of the fork-join model Synchronization cost due to implicit barriers Under-exploitation of potential parallelism Poor data locality and no inter-statement optimization 26 of 41
From data to task parallelism Antoine Tran Tan PHD, 2012-2015 Limits of the fork-join model Synchronization cost due to implicit barriers Under-exploitation of potential parallelism Poor data locality and no inter-statement optimization Skeletons from the Future Adapt current skeletons for taskication Use Futures ( or HPX) to automatically pipeline Derive a dependency graph between statements 26 of 41
Parallel Skeletons extraction process - Take 2 A = B / sum(C+D); ; ; = tmp sum + C D fold = A / B tmp transform 27 of 41
Parallel Skeletons extraction process - Take 2 A = B / sum(C+D); fold = tmp(3) sum(3) + C(:, 3) D(:, 3) transform = A(:, 3) / B(:, 3) tmp(3) fold = tmp(2) sum(2) + C(:, 2) D(:, 2) transform = A(:, 2) / B(:, 2) tmp(2) workerfold,simd = tmp(1) sum + C(:, 1) D(:, 1) workertransform,simd = A(:, 1) / B(:, 1) tmp(1) spawnertransform,OpenMP spawnertransform,OpenMP ; 28 of 41
Motion Detection Lacassagne et al., ICIP 2009 Sigma-Delta algorithm based on background substraction Use local gaussian model of lightness variation to detect motion Challenge: Very low arithmetic density Challenge: Integer-based implementation with small range 29 of 41
Motion Detection table <char > sigma_delta( table <char >& background , table <char > const& frame , table <char >& variance ) { // Estimate Raw Movement background = selinc( background < frame , seldec(background > frame , background) ); table <char > diff = dist(background , frame); // Compute Local Variance table <char > sig3 = muls(diff ,3); var = if_else( diff != 0 , selinc( variance < sig3 , seldec( var > sig3 , variance) ) , variance ); // Generate Movement Label return if_zero_else_one( diff < variance ); } 30 of 41
Motion Detection 0 2 4 6 8 10 12 14 16 18 512x512 1024x1024 cycles/element Image Size (N x N) x6.8 x14.8 x16.5 x2.1 x3.6 x6.7 x15.3 x18 x2.3 x3.99 x10.8 x10.8 SCALAR HALF CORE FULL CORE SIMD JRTIP2008 SIMD + HALF CORE SIMD + FULL CORE 31 of 41
Black and Scholes Option Pricing NT2 Code table <float > blackscholes( table <float > const& Sa, table <float > const& Xa , table <float > const& Ta , table <float > const& ra, table <float > const& va ) { table <float > da = sqrt(Ta); table <float > d1 = log(Sa/Xa) + (sqr(va)*0.5f+ra)*Ta/(va*da); table <float > d2 = d1 -va*da; return Sa*normcdf(d1)- Xa*exp(-ra*Ta)*normcdf(d2); } 32 of 41
Black and Scholes Option Pricing NT2 Code with loop fusion table <float > blackscholes( table <float > const& Sa, table <float > const& Xa , table <float > const& Ta , table <float > const& ra, table <float > const& va ) { // Preallocate temporary tables table <float > da(extent(Ta)), d1(extent(Ta)), d2(extent(Ta)), R(extent(Ta)); // tie merge loop nest and increase cache locality tie(da,d1,d2,R) = tie( sqrt(Ta) , log(Sa/Xa) + (sqr(va)*0.5f+ra)*Ta/(va*da) , d1 -va*da , Sa*normcdf(d1)- Xa*exp(-ra*Ta)*normcdf(d2) ); return R; } 32 of 41
Black and Scholes Option Pricing Performance 1000000 0 50 100 150 x1.89 x2.91 x5.58 x6.30 Size cycle/value scalar SSE2 AVX2 SSE2, 4 cores AVX2, 4 cores 33 of 41
Black and Scholes Option Pricing Performance with loop fusion/futurisation 1000000 0 50 100 150 x2.27 x4.13 x8.05 x11.12 Size cycle/value scalar SSE2 AVX2 SSE2, 4 cores AVX2, 4 cores 34 of 41
LU Decomposition Algorithm A00 A01 A02A10 A20 A11 A21 A12 A22 A11 A12A21 A22 A22 step 1 step 2 step 3 step 4 step 5 step 6 step 7 DGETRF DGESSM DTSTRF DSSSSM 35 of 41
LU Decomposition Performance 0 10 20 30 40 50 0 50 100 Number of cores MedianGFLOPS 8000 × 8000 LU decomposition NT2 Intel MKL 36 of 41
Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 37 of 41
Conclusion Parallel Computing for Scientist Software Libraries built as Generic and Generative components can solve a large chunk of parallelism related problems while being easy to use. Like regular language, DSEL needs informations about the hardware system Integrating hardware descriptions as Generic components increases tools portability and re-targetability 38 of 41
Conclusion Parallel Computing for Scientist Software Libraries built as Generic and Generative components can solve a large chunk of parallelism related problems while being easy to use. Like regular language, DSEL needs informations about the hardware system Integrating hardware descriptions as Generic components increases tools portability and re-targetability Our Achievements A new method for parallel software development Efficient libraries working on large subset of hardware High level of performances across a wide application spectrum 38 of 41
Works in Progress Application to Accelerators Exploration of proper skeleton implementation on GPUs Adaptation of Future based code generator In progress with Ian Masliah’ PHD thesis Parallelism within C++ SIMD as part of the standard library Proposal N3571 for standard SIMD computation Interoperability with current parallel model of C++ 39 of 41
Perspectives DSEL as C++ rst class idiom Build partial evaluation into the language Ease transition between regular and meta C++ Mid-term Prospect: metaOCAML like quoting for C++ DSEL and compilers relationship C++ DSEL hits a limit on their applicability Compilers often lack high level informations for proper optimization Mid-term Prospect: Hybrid library/compiler approaches for DSEL 40 of 41
Thanks for your attention

More Related Content

PDF
(Costless) Software Abstractions for Parallel Architectures
Joel Falcou
 
PDF
Boost.Dispatch
Joel Falcou
 
PDF
Automatic Task-based Code Generation for High Performance DSEL
Joel Falcou
 
PDF
Designing Architecture-aware Library using Boost.Proto
Joel Falcou
 
PDF
Software Abstractions for Parallel Hardware
Joel Falcou
 
PPTX
C programming interview questions
adarshynl
 
PPT
C++ OOP Implementation
Fridz Felisco
 
PPT
OOP in C++
ppd1961
 
(Costless) Software Abstractions for Parallel Architectures
Joel Falcou
 
Boost.Dispatch
Joel Falcou
 
Automatic Task-based Code Generation for High Performance DSEL
Joel Falcou
 
Designing Architecture-aware Library using Boost.Proto
Joel Falcou
 
Software Abstractions for Parallel Hardware
Joel Falcou
 
C programming interview questions
adarshynl
 
C++ OOP Implementation
Fridz Felisco
 
OOP in C++
ppd1961
 

What's hot (20)

PDF
Generating Assertion Code from OCL: A Transformational Approach Based on Simi...
Institute of Science Tokyo
 
PDF
The Goal and The Journey - Turning back on one year of C++14 Migration
Joel Falcou
 
PDF
Top C Language Interview Questions and Answer
Vineet Kumar Saini
 
DOCX
Vb.net
priyadharshini murugan
 
PDF
A recommender system for generalizing and refining code templates
Coen De Roover
 
PDF
C++ questions And Answer
lavparmar007
 
DOCX
PHP
priyadharshini murugan
 
PPT
C++ Interview Questions
Kaushik Raghupathi
 
PPT
C, C++ Interview Questions Part - 1
ReKruiTIn.com
 
PDF
C++ interview question
Durgesh Tripathi
 
PPT
Intro. to prog. c++
KurdGul
 
PDF
Compiler Construction | Lecture 6 | Introduction to Static Analysis
Eelco Visser
 
PPT
01 c++ Intro.ppt
Tareq Hasan
 
PPT
C by balaguruswami - e.balagurusamy
Srichandan Sobhanayak
 
PDF
A Recommender System for Refining Ekeko/X Transformation
Coen De Roover
 
PPT
C++ questions and answers
Deepak Singh
 
PDF
Implementation
adil raja
 
PDF
C programming & data structure [character strings & string functions]
MomenMostafa
 
PPT
C programming for Computing Techniques
Appili Vamsi Krishna
 
PDF
Deep C
Olve Maudal
 
Generating Assertion Code from OCL: A Transformational Approach Based on Simi...
Institute of Science Tokyo
 
The Goal and The Journey - Turning back on one year of C++14 Migration
Joel Falcou
 
Top C Language Interview Questions and Answer
Vineet Kumar Saini
 
Vb.net
priyadharshini murugan
 
A recommender system for generalizing and refining code templates
Coen De Roover
 
C++ questions And Answer
lavparmar007
 
PHP
priyadharshini murugan
 
C++ Interview Questions
Kaushik Raghupathi
 
C, C++ Interview Questions Part - 1
ReKruiTIn.com
 
C++ interview question
Durgesh Tripathi
 
Intro. to prog. c++
KurdGul
 
Compiler Construction | Lecture 6 | Introduction to Static Analysis
Eelco Visser
 
01 c++ Intro.ppt
Tareq Hasan
 
C by balaguruswami - e.balagurusamy
Srichandan Sobhanayak
 
A Recommender System for Refining Ekeko/X Transformation
Coen De Roover
 
C++ questions and answers
Deepak Singh
 
Implementation
adil raja
 
C programming & data structure [character strings & string functions]
MomenMostafa
 
C programming for Computing Techniques
Appili Vamsi Krishna
 
Deep C
Olve Maudal
 
Ad

Similar to HDR Defence - Software Abstractions for Parallel Architectures (20)

PDF
main
Ian Masliah
 
PPTX
Data-Centric Parallel Programming
inside-BigData.com
 
PPTX
Heterogeneous programming
Boyana Norris
 
PDF
Массовый параллелизм для гетерогенных вычислений на C++ для беспилотных автом...
CEE-SEC(R)
 
PDF
Data Analytics and Simulation in Parallel with MATLAB*
Intel® Software
 
PPTX
Thinking in parallel ab tuladev
Pavel Tsukanov
 
PDF
Application Optimisation using OpenPOWER and Power 9 systems
Ganesan Narayanasamy
 
PDF
Ndp Slides
oscon2007
 
PDF
Ehsan parallel accelerator-dec2015
Christian Peel
 
PPTX
Abstractions and Directives for Adapting Wavefront Algorithms to Future Archi...
inside-BigData.com
 
PDF
infoShare AI Roadshow 2018 - Tomasz Kopacz (Microsoft) - jakie możliwości daj...
Infoshare
 
PDF
ExaLearn Overview - ECP Co-Design Center for Machine Learning
inside-BigData.com
 
PDF
Directive-based approach to Heterogeneous Computing
Ruymán Reyes
 
PPT
Tim Popl
mchaar
 
PPT
The Next Mainstream Programming Language: A Game Developer’s Perspective
guest4fd7a2
 
PDF
Assisting User’s Transition to Titan’s Accelerated Architecture
inside-BigData.com
 
PPTX
SequenceL Auto-Parallelizing Toolset Intro slideshare
Doug Norton
 
PPTX
SequenceL intro slideshare
Doug Norton
 
PDF
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
PPSX
Research Scope in Parallel Computing And Parallel Programming
Shitalkumar Sukhdeve
 
main
Ian Masliah
 
Data-Centric Parallel Programming
inside-BigData.com
 
Heterogeneous programming
Boyana Norris
 
Массовый параллелизм для гетерогенных вычислений на C++ для беспилотных автом...
CEE-SEC(R)
 
Data Analytics and Simulation in Parallel with MATLAB*
Intel® Software
 
Thinking in parallel ab tuladev
Pavel Tsukanov
 
Application Optimisation using OpenPOWER and Power 9 systems
Ganesan Narayanasamy
 
Ndp Slides
oscon2007
 
Ehsan parallel accelerator-dec2015
Christian Peel
 
Abstractions and Directives for Adapting Wavefront Algorithms to Future Archi...
inside-BigData.com
 
infoShare AI Roadshow 2018 - Tomasz Kopacz (Microsoft) - jakie możliwości daj...
Infoshare
 
ExaLearn Overview - ECP Co-Design Center for Machine Learning
inside-BigData.com
 
Directive-based approach to Heterogeneous Computing
Ruymán Reyes
 
Tim Popl
mchaar
 
The Next Mainstream Programming Language: A Game Developer’s Perspective
guest4fd7a2
 
Assisting User’s Transition to Titan’s Accelerated Architecture
inside-BigData.com
 
SequenceL Auto-Parallelizing Toolset Intro slideshare
Doug Norton
 
SequenceL intro slideshare
Doug Norton
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
inside-BigData.com
 
Research Scope in Parallel Computing And Parallel Programming
Shitalkumar Sukhdeve
 
Ad

Recently uploaded (20)

PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
PPTX
Introduction to Artificial Intelligence
lohedi9363
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PDF
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
PPTX
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
PPTX
Reimagine Home Health with the Power of Agentic AI​
AutomationEdge Technologies
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
Introduction to Artificial Intelligence
lohedi9363
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
L1 - Introduction to python Backend.pptx
MoizAhmed480282
 
Reimagine Home Health with the Power of Agentic AI​
AutomationEdge Technologies
 

HDR Defence - Software Abstractions for Parallel Architectures

  • 1. Software Abstractions for Parallel Architectures Joel Falcou LRI - CNRS - INRIA HDR Thesis Defense 12/01/2014
  • 2. The Paradigm Change in Science From Experiments to Simulations Simulations is now an integral part of the Scientic Method Scientic Computing enables larger, faster, more accurate Research Fast Simulation is Time Travel as scientic results are now more readily available Local Galaxy Cluster Simulation - Illustris project Computing is rst and foremost a mainstream science tool 2 of 41
  • 3. The Paradigm Change in Science The Parallel Hell Heat Wall: Growing cores instead of GHz Hierarchical and heterogeneous parallel systems are the norm The Free Lunch is over as hardware complexity rises faster than the average developer skills Local Galaxy Cluster Simulation - Illustris project The real challenge in HPC is the Expressiveness/Efficiency War 2 of 41
  • 4. The Expressiveness/Efficiency War Single Core Era Performance Expressiveness C/Fort. C++ Java Multi-Core/SIMD Era Performance Expressiveness Sequential Threads SIMD Heterogenous Era Performance Expressiveness Sequential SIMD Threads GPU Phi Distributed As parallel systems complexity grows, the expressiveness gap turns into an ocean 3 of 41
  • 5. Designing tools for Scientic Computing Objectives 1. Be non-disruptive 2. Domain driven optimizations 3. Provide intuitive API for the user 4. Support a wide architectural landscape 5. Be efficient 4 of 41
  • 6. Designing tools for Scientic Computing Objectives 1. Be non-disruptive 2. Domain driven optimizations 3. Provide intuitive API for the user 4. Support a wide architectural landscape 5. Be efficient Our Approach Design tools as C++ libraries (1) Design these libraries as Domain Specic Embedded Languages (DSEL) (2+3) Use Parallel Programming Abstractions as parallel components (4) Use Generative Programming to deliver performance (5) 4 of 41
  • 7. Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 5 of 41
  • 8. Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap 6 of 41
  • 9. Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap Available Models Performance centric: P-RAM, LOG-P, BSP Pattern centric: Futures, Skeletons Data centric: HTA, PGAS 6 of 41
  • 10. Why Parallel Programming Models ? Limits of regular tools Unstructured parallelism is error-prone Low level parallel tools are non-composable Contribute to the Expressiveness Gap Available Models Performance centric: P-RAM, LOG-P, BSP Pattern centric: Futures, Skeletons Data centric: HTA, PGAS 6 of 41
  • 11. Bulk Synchronous Parallelism [Valiant, McColl 90] Principles Machine Model Execution Model Analytic Cost Model C o m p u t e B a r r i e r C o m m Wmax h.g P0 P1 P2 P3 Superstep T Superstep T+1 LWmax h.g BSP Execution Model 7 of 41
  • 12. Bulk Synchronous Parallelism [Valiant, McColl 90] Advantages Simple set of primitives Implementable on any kind of hardware Possibility to reason about BSP programs C o m p u t e B a r r i e r C o m m Wmax h.g P0 P1 P2 P3 Superstep T Superstep T+1 LWmax h.g BSP Execution Model 7 of 41
  • 13. Parallel Skeletons [Cole 89] Principles There are patterns in parallel applications Those patterns can be generalized in Skeletons Applications are assembled as a combination of such patterns Functional point of view Skeletons are Higher-Order Functions Skeletons support a compositionnal semantic Applications become composition of state-less functions 8 of 41
  • 14. Parallel Skeletons [Cole 89] Principles There are patterns in parallel applications Those patterns can be generalized in Skeletons Applications are assembled as a combination of such patterns Classical Skeletons Data parallel: map, fold, scan Task parallel: par, pipe, farm More complex: Distribuable Homomorphism, Divide & Conquer, … 8 of 41
  • 15. Relevance to our Objectives Why using Parallel Skeletons ? Write code independant of parallel programming minutiae Composability supports hierarchical architectures Code is scalable and easy to maintain Why using BSP ? Cost model guide development Few primitives mean that intellectual burden is low Good medium for developping skeletons How to ensure performance of those models’ implementations ? 9 of 41
  • 16. Domain Specic Embedded Languages Domain Specic Languages Non-Turing complete declarative languages Solve a single type of problems Express what to do instead of how to do it E.g: SQL, M, M, … From DSL to DSEL [Abrahams 2004] A DSL incorporates domain-specic notation, constructs, and abstractions as fundamental design considerations. A Domain Specic Embedded Languages (DSEL) is simply a library that meets the same criteria Generative Programming is one way to design such libraries 10 of 41
  • 17. Generative Programming [Eisenecker 97] Domain Specific Application Description Generative Component Concrete Application Translator Parametric Sub-components 11 of 41
  • 18. Meta-programming as a Tool Denition Meta-programming is the writing of computer programs that analyse, transform and generate other programs (or themselves) as their data. Meta-programmable Languages metaOCAML : runtime code generation via code quoting Template Haskell : compile-time code generation via templates C++ : compile-time code generation via templates C++ meta-programming Relies on the Turing-complete C++  sub-language Handles types and integral constants at compile-time  classes and functions act as code quoting 12 of 41
  • 19. The Expression Templates Idiom Principles Relies on extensive operator overloading Carries semantic information around code fragment Introduces DSLs without disrupting dev. chain matrix x(h,w),a(h,w),b(h,w); x = cos(a) + (b*a); expr<assign ,expr<matrix&> ,expr<plus , expr<cos ,expr<matrix&> > , expr<multiplies ,expr<matrix&> ,expr<matrix&> > >(x,a,b); + *cos a ab = x #pragma omp parallel for for(int j=0;j<h;++j) { for(int i=0;i<w;++i) { x(j,i) = cos(a(j,i)) + ( b(j,i) * a(j,i) ); } } Arbitrary Transforms applied on the meta-AST General Principles of Expression Templates 13 of 41
  • 20. The Expression Templates Idiom Advantages Generic implementation becomes self-aware of optimizations API abstraction level is arbitrary high Accessible through high-level tools like B.P matrix x(h,w),a(h,w),b(h,w); x = cos(a) + (b*a); expr<assign ,expr<matrix&> ,expr<plus , expr<cos ,expr<matrix&> > , expr<multiplies ,expr<matrix&> ,expr<matrix&> > >(x,a,b); + *cos a ab = x #pragma omp parallel for for(int j=0;j<h;++j) { for(int i=0;i<w;++i) { x(j,i) = cos(a(j,i)) + ( b(j,i) * a(j,i) ); } } Arbitrary Transforms applied on the meta-AST General Principles of Expression Templates 13 of 41
  • 21. Our Contributions Our Strategy Applies DSEL generation techniques to parallel programming Maintains low cost of abstractions through meta-programming Maintains abstraction level via modern library design Our contributions Tools Pub. Scope Applications Quaff ParCo’06 MPI Skeletons Real-time 3D reconstruction SkellBE PACT’08 Skeleton on Cell BE Real-time Image processing BSP++ IJPP’12 MPI/OpenMP BSP Bioinformatics, Model Checking NT2 JPDC’14 Data Parallel Matlab Fluid Dynamics, Vision 14 of 41
  • 22. Example of BSP++ Application Khaled Hamidouche PHD 2008-2011 in collab. with Univ. Brasilia BSP Smith & Waterman S&W computes DNA sequences alignment BSP++ implementation was written once and run on 7 different hardwares Efficiency of 95+% even on 6000 cores super-computer Platform MaxSize # Elements Speedup GCUPs cluster (MPI) 1,072,950 128 cores 73x 6.53 cluster (MPI/OpenMP) 1,072,950 128 cores 116x 10.41 OpenMP 1,072,950 16 cores 16x 0.40 CellBE 85,603 8 SPEs — 0.14 cluster of CellBEs 85,603 24 SPEs (8:24) 2.8x 0.37 Hopper(MPI) 5,303,436 3072 cores 260x 3.09 Hopper(MPI+OpenMP) 24,894,269 6144 cores 5664x 15,5 15 of 41
  • 23. Second Look at our Contributions Development Limitations DSELs are mostly tied to the domain model Architecture support is often an afterthought Extensibility is difficult as many refactoring are required per architecture Example : No proper way to support GPUs with those implementation techniques 16 of 41
  • 24. Second Look at our Contributions Development Limitations DSELs are mostly tied to the domain model Architecture support is often an afterthought Extensibility is difficult as many refactoring are required per architecture Example : No proper way to support GPUs with those implementation techniques Proposed Method Extends Generative Programming to take this architecture into account Provides an architecture description DSEL Integrates this description in the code generation process 16 of 41
  • 25. Architecture Aware Generative Programming 17 of 41
  • 26. Software refactoring Tools Issues Changes Quaff Raw skeletons API Re-engineered as part of NT2 SkellBE Too architecture specic Re-engineered as part of NT2 BSP++ Integration issues Integrate hybrid code generation NT2 Not easily extendable Integrate Quaff Skeleton models Boost.SIMD - Side product of NT2 restructuration Conclusion Skeletons are ne as parallel middleware Model based abstractions are not high level enough For low level architectures, the simplest model is often the best 18 of 41
  • 27. Boost.SIMD Pierre Estérie PHD 2010-2014 Principles Provides simple C++ API over SIMD extensions Supports every Intel, PPC and ARM instructions sets Fully integrates with modern C++ idioms Sparse Tridiagonal Solver - collaboration with M. Baboulin and Y. wang 19 of 41
  • 28. Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 20 of 41
  • 29. The Numerical Template Toolbox Pierre Estérie PHD 2010-2014 NT2 as a Scientic Computing Library Provides a simple, M-like interface for users Provides high-performance computing entities and primitives Is easily extendable Components Uses Boost.SIMD for in-core optimizations Uses recursive parallel skeletons Supports task parallelism through Futures 21 of 41
  • 30. The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M 22 of 41
  • 31. The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le 22 of 41
  • 32. The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le Add #include <nt2/nt2.hpp> and do cosmetic changes 22 of 41
  • 33. The Numerical Template Toolbox Principles table<T,S> is a simple, multidimensional array object that exactly mimics M array behavior and functionalities 500+ functions usable directly either on table or on any scalar values as in M How does it works Take a .m le, copy to a .cpp le Add #include <nt2/nt2.hpp> and do cosmetic changes Compile the le and link with libnt2.a 22 of 41
  • 34. NT2 - From M to C++ M code A1 = 1:1000; A2 = A1 + randn(size(A1)); X = lu(A1*A1’); rms = sqrt( sum(sqr(A1(:) - A2(:))) / numel(A1) ); NT2 code table <double > A1 = _(1. ,1000.); table <double > A2 = A1 + randn(size(A1)); table <double > X = lu( mtimes(A1, trans(A1) ); double rms = sqrt( sum(sqr(A1(_) - A2(_))) / numel(A1) ); 23 of 41
  • 35. Parallel Skeletons extraction process A = B / sum(C+D); = A / B sum + C D fold transform 24 of 41
  • 36. Parallel Skeletons extraction process A = B / sum(C+D); ; ; = A / B sum + C D fold transform = tmp sum + C D fold ⇒ = A / B tmp transform 25 of 41
  • 37. From data to task parallelism Antoine Tran Tan PHD, 2012-2015 Limits of the fork-join model Synchronization cost due to implicit barriers Under-exploitation of potential parallelism Poor data locality and no inter-statement optimization 26 of 41
  • 38. From data to task parallelism Antoine Tran Tan PHD, 2012-2015 Limits of the fork-join model Synchronization cost due to implicit barriers Under-exploitation of potential parallelism Poor data locality and no inter-statement optimization Skeletons from the Future Adapt current skeletons for taskication Use Futures ( or HPX) to automatically pipeline Derive a dependency graph between statements 26 of 41
  • 39. Parallel Skeletons extraction process - Take 2 A = B / sum(C+D); ; ; = tmp sum + C D fold = A / B tmp transform 27 of 41
  • 40. Parallel Skeletons extraction process - Take 2 A = B / sum(C+D); fold = tmp(3) sum(3) + C(:, 3) D(:, 3) transform = A(:, 3) / B(:, 3) tmp(3) fold = tmp(2) sum(2) + C(:, 2) D(:, 2) transform = A(:, 2) / B(:, 2) tmp(2) workerfold,simd = tmp(1) sum + C(:, 1) D(:, 1) workertransform,simd = A(:, 1) / B(:, 1) tmp(1) spawnertransform,OpenMP spawnertransform,OpenMP ; 28 of 41
  • 41. Motion Detection Lacassagne et al., ICIP 2009 Sigma-Delta algorithm based on background substraction Use local gaussian model of lightness variation to detect motion Challenge: Very low arithmetic density Challenge: Integer-based implementation with small range 29 of 41
  • 42. Motion Detection table <char > sigma_delta( table <char >& background , table <char > const& frame , table <char >& variance ) { // Estimate Raw Movement background = selinc( background < frame , seldec(background > frame , background) ); table <char > diff = dist(background , frame); // Compute Local Variance table <char > sig3 = muls(diff ,3); var = if_else( diff != 0 , selinc( variance < sig3 , seldec( var > sig3 , variance) ) , variance ); // Generate Movement Label return if_zero_else_one( diff < variance ); } 30 of 41
  • 43. Motion Detection 0 2 4 6 8 10 12 14 16 18 512x512 1024x1024 cycles/element Image Size (N x N) x6.8 x14.8 x16.5 x2.1 x3.6 x6.7 x15.3 x18 x2.3 x3.99 x10.8 x10.8 SCALAR HALF CORE FULL CORE SIMD JRTIP2008 SIMD + HALF CORE SIMD + FULL CORE 31 of 41
  • 44. Black and Scholes Option Pricing NT2 Code table <float > blackscholes( table <float > const& Sa, table <float > const& Xa , table <float > const& Ta , table <float > const& ra, table <float > const& va ) { table <float > da = sqrt(Ta); table <float > d1 = log(Sa/Xa) + (sqr(va)*0.5f+ra)*Ta/(va*da); table <float > d2 = d1 -va*da; return Sa*normcdf(d1)- Xa*exp(-ra*Ta)*normcdf(d2); } 32 of 41
  • 45. Black and Scholes Option Pricing NT2 Code with loop fusion table <float > blackscholes( table <float > const& Sa, table <float > const& Xa , table <float > const& Ta , table <float > const& ra, table <float > const& va ) { // Preallocate temporary tables table <float > da(extent(Ta)), d1(extent(Ta)), d2(extent(Ta)), R(extent(Ta)); // tie merge loop nest and increase cache locality tie(da,d1,d2,R) = tie( sqrt(Ta) , log(Sa/Xa) + (sqr(va)*0.5f+ra)*Ta/(va*da) , d1 -va*da , Sa*normcdf(d1)- Xa*exp(-ra*Ta)*normcdf(d2) ); return R; } 32 of 41
  • 46. Black and Scholes Option Pricing Performance 1000000 0 50 100 150 x1.89 x2.91 x5.58 x6.30 Size cycle/value scalar SSE2 AVX2 SSE2, 4 cores AVX2, 4 cores 33 of 41
  • 47. Black and Scholes Option Pricing Performance with loop fusion/futurisation 1000000 0 50 100 150 x2.27 x4.13 x8.05 x11.12 Size cycle/value scalar SSE2 AVX2 SSE2, 4 cores AVX2, 4 cores 34 of 41
  • 48. LU Decomposition Algorithm A00 A01 A02A10 A20 A11 A21 A12 A22 A11 A12A21 A22 A22 step 1 step 2 step 3 step 4 step 5 step 6 step 7 DGETRF DGESSM DTSTRF DSSSSM 35 of 41
  • 49. LU Decomposition Performance 0 10 20 30 40 50 0 50 100 Number of cores MedianGFLOPS 8000 × 8000 LU decomposition NT2 Intel MKL 36 of 41
  • 50. Talk Layout Introduction Abstractions & Efficiency Experimental Results Conclusion 37 of 41
  • 51. Conclusion Parallel Computing for Scientist Software Libraries built as Generic and Generative components can solve a large chunk of parallelism related problems while being easy to use. Like regular language, DSEL needs informations about the hardware system Integrating hardware descriptions as Generic components increases tools portability and re-targetability 38 of 41
  • 52. Conclusion Parallel Computing for Scientist Software Libraries built as Generic and Generative components can solve a large chunk of parallelism related problems while being easy to use. Like regular language, DSEL needs informations about the hardware system Integrating hardware descriptions as Generic components increases tools portability and re-targetability Our Achievements A new method for parallel software development Efficient libraries working on large subset of hardware High level of performances across a wide application spectrum 38 of 41
  • 53. Works in Progress Application to Accelerators Exploration of proper skeleton implementation on GPUs Adaptation of Future based code generator In progress with Ian Masliah’ PHD thesis Parallelism within C++ SIMD as part of the standard library Proposal N3571 for standard SIMD computation Interoperability with current parallel model of C++ 39 of 41
  • 54. Perspectives DSEL as C++ rst class idiom Build partial evaluation into the language Ease transition between regular and meta C++ Mid-term Prospect: metaOCAML like quoting for C++ DSEL and compilers relationship C++ DSEL hits a limit on their applicability Compilers often lack high level informations for proper optimization Mid-term Prospect: Hybrid library/compiler approaches for DSEL 40 of 41
  • 55. Thanks for your attention