Parallel Programming
3 likes2,040 views
This document discusses parallel computing and summarizes a talk on the topic. It begins by noting the challenge of exploiting parallelism in hardware to solve problems more effectively. It then discusses single-core CPUs and how multi-core architectures allow multiple cores to run threads in parallel. Within each core, threads are time-sliced. Instruction-level parallelism and thread-level parallelism are described. Simultaneous multithreading is presented as allowing multiple independent threads to execute simultaneously on the same core. However, SMT does not provide true parallelism like multi-core architectures, which allow threads to run independently on separate cores each with their own resources.
![Instruction level parallelism
• For(int i-0;i<1000;i++)
{ a[0]++; a[0]++; }
• For(int i-0;i<1000;i++)
{ a[0]++; a[1]++; }](https://image.slidesharecdn.com/ot4-parallelprogramming-sundararajans-sep29-120103020019-phpapp01/85/Parallel-Programming-10-320.jpg)
Parallel Programming
- 1. Learning and Development Presents OPEN TALK SERIES A series of illuminating talks and interactions that open our minds to new ideas and concepts; that makes us look for newer or better ways of doing what we did; or point us to exciting things we have never done before. A range of topics on Technology, Business, Fun and Life. Be part of the learning experience at Aditi. Join the talks. Its free. Free as in freedom at work, not free-beer. Speak at these events. Or bring an expert/friend to talk. Mail LEAD with topic and availability.
- 2. Parallel Programming Sundararajan Subramanian Aditi Technologies 2
- 3. Introduction to Parallel Computing • The challenge – Provide the abstractions , programming paradigms, and algorithms needed to effectively design, implement, and maintain applications that exploit the parallelism provided by the underlying hardware in order to solve modern problems.
- 4. Single-core CPU chip the single core 4
- 5. Multi-core architectures Core 1 Core 2 Core 3 Core 4 Multi-core CPU chip 5
- 6. Multi-core CPU chip • The cores fit on a single processor socket • Also called CMP (Chip Multi-Processor) c c c c o o o o r r r r e e e e 1 2 3 4 6
- 7. The cores run in parallel thread 1 thread 2 thread 3 thread 4 c c c c o o o o r r r r e e e e 1 2 3 4 7
- 8. Within each core, threads are time-sliced (just like on a uniprocessor) several several several several threads threads threads threads c c c c o o o o r r r r e e e e 1 2 3 4 8
- 9. Instruction-level parallelism • Parallelism at the machine-instruction level • The processor can re-order, pipeline instructions, split them into microinstructions, do aggressive branch prediction, etc. • Instruction-level parallelism enabled rapid increases in processor speeds over the last 15 years 9
- 10. Instruction level parallelism • For(int i-0;i<1000;i++) { a[0]++; a[0]++; } • For(int i-0;i<1000;i++) { a[0]++; a[1]++; }
- 11. Thread-level parallelism (TLP) • This is parallelism on a more coarser scale • Server can serve each client in a separate thread (Web server, database server) • A computer game can do AI, graphics, and physics in three separate threads • Single-core superscalar processors cannot fully exploit TLP • Multi-core architectures are the next step in processor evolution: explicitly exploiting TLP 11
- 12. A technique complementary to multi-core: Simultaneous multithreading • Problem addressed: L1 D-Cache D-TLB The processor pipeline Integer Floating Point can get stalled: L2 Cache and Control – Waiting for the result Schedulers of a long floating point Uop queues (or integer) operation Rename/Alloc – Waiting for data to BTB Trace Cache uCode arrive from memory ROM Decoder Other execution units Bus wait unused BTB and I-TLB Source: Intel 12
- 13. Simultaneous multithreading (SMT) • Permits multiple independent threads to execute SIMULTANEOUSLY on the SAME core • Weaving together multiple “threads” on the same core • Example: if one thread is waiting for a floating point operation to complete, another thread can use the integer units 13
- 14. Without SMT, only a single thread can run at any given time L1 D-Cache D-TLB L2 Cache and Control Integer Floating Point Schedulers Uop queues Rename/Alloc BTB Trace Cache uCode ROM Decoder Bus BTB and I-TLB Thread 1: floating point 14
- 15. Without SMT, only a single thread can run at any given time L1 D-Cache D-TLB L2 Cache and Control Integer Floating Point Schedulers Uop queues Rename/Alloc BTB Trace Cache uCode ROM Decoder Bus BTB and I-TLB Thread 2: integer operation 15
- 16. SMT processor: both threads can run concurrently L1 D-Cache D-TLB L2 Cache and Control Integer Floating Point Schedulers Uop queues Rename/Alloc BTB Trace Cache uCode ROM Decoder Bus BTB and I-TLB Thread 2: Thread 1: floating point integer operation 16
- 17. But: Can’t simultaneously use the same functional unit L1 D-Cache D-TLB L2 Cache and Control Integer Floating Point Schedulers Uop queues Rename/Alloc BTB Trace Cache uCode ROM Decoder This scenario is impossible with SMT Bus BTB and I-TLB on a single core Thread 1 Thread 2 (assuming a single IMPOSSIBLE integer unit) 17
- 18. SMT not a “true” parallel processor • Enables better threading (e.g. up to 30%) • OS and applications perceive each simultaneous thread as a separate “virtual processor” • The chip has only a single copy of each resource • Compare to multi-core: each core has its own copy of resources 18
- 19. Multi-core: threads can run on separate cores L1 D-Cache D-TLB L1 D-Cache D-TLB Integer Floating Point Integer Floating Point L2 Cache and Control L2 Cache and Control Schedulers Schedulers Uop queues Uop queues Rename/Alloc Rename/Alloc BTB Trace Cache uCode BTB Trace Cache uCode ROM ROM Decoder Decoder Bus Bus BTB and I-TLB BTB and I-TLB Thread 1 Thread 2 19
- 20. Multi-core: threads can run on separate cores L1 D-Cache D-TLB L1 D-Cache D-TLB Integer Floating Point Integer Floating Point L2 Cache and Control L2 Cache and Control Schedulers Schedulers Uop queues Uop queues Rename/Alloc Rename/Alloc BTB Trace Cache uCode BTB Trace Cache uCode ROM ROM Decoder Decoder Bus Bus BTB and I-TLB BTB and I-TLB Thread 3 Thread 4 20
- 21. Combining Multi-core and SMT • Cores can be SMT-enabled (or not) • The different combinations: – Single-core, non-SMT: standard uniprocessor – Single-core, with SMT – Multi-core, non-SMT – Multi-core, with SMT: our fish machines • The number of SMT threads: 2, 4, or sometimes 8 simultaneous threads • Intel calls them “hyper-threads” 21
- 22. SMT Dual-core: all four threads can run concurrently L1 D-Cache D-TLB L1 D-Cache D-TLB Integer Floating Point Integer Floating Point L2 Cache and Control L2 Cache and Control Schedulers Schedulers Uop queues Uop queues Rename/Alloc Rename/Alloc BTB Trace Cache uCode BTB Trace Cache uCode ROM ROM Decoder Decoder Bus Bus BTB and I-TLB BTB and I-TLB Thread 1 Thread 3 Thread 2 Thread 4 22
- 24. Designs with private L2 caches CORE0 CORE1 CORE0 CORE1 L1 cache L1 cache L1 cache L1 cache L2 cache L2 cache L2 cache L2 cache L3 cache L3 cache memory memory Both L1 and L2 are private A design with L3 caches Examples: AMD Opteron, AMD Athlon, Intel Pentium D Example: Intel Itanium 2
- 25. Private vs shared caches? • Advantages/disadvantages? 25
- 26. Private vs shared caches • Advantages of private: – They are closer to core, so faster access – Reduces contention • Advantages of shared: – Threads on different cores can share the same cache data – More cache space available if a single (or a few) high-performance thread runs on the system 26
- 27. Parallel Architectures • Use multiple – Datapaths – Memory units – Processing units
- 28. Parallel Architectures • SIMD – Single instruction stream, multiple data stream Processing Unit Processing Unit Interconnect Control Processing Unit Unit Processing Unit Processing Unit
- 29. Parallel Architectures • MIMD – Multiple instruction stream, multiple data stream Processing/Control Unit Processing/Control Interconnect Unit Processing/Control Unit Processing/Control Unit
- 30. Parallelism in Visual Studio 2010 Integrated Programming Models Programming Models Tooling PLINQ Parallel Task Parallel Parallel Pattern Agents Debugger Library Library Library Toolwindows Data Structures Data Structures Concurrency Runtime Concurrency Runtime ThreadPool Profiler Task Scheduler Concurrency Task Scheduler Analysis Resource Manager Resource Manager Operating System Threads Key: Tools Native Library Managed Library
- 31. Multi threading Today • Divide the total number of activites across n processors • In case of 2 Procs, divide it by 2.
- 32. User Mode Scheduler CLR Thread Pool Global Queue Worker … Worker Thread 1 Thread p Program Thread
- 33. User Mode Scheduler For Tasks CLR Thread Pool: Work-Stealing Local … Local Global Queue Queue Queue Worker … Worker Thread 1 Thread p Task 6 Task 1 Task Task 3 4 Task 2Program Task 5 Thread
- 34. DEMO
- 35. Task-based Programming ThreadPool Summary ThreadPool.QueueUserWorkItem(…); System.Threading.Tasks Starting Parent/Child Task.Factory.StartNew(…); var p = new Task(() => { var t = new Task(…); }); Continue/Wait/Cancel Task t = … Tasks with results Task<int> f = Task p = t.ContinueWith(…); new Task<int>(() => C()); t.Wait(2000); … t.Cancel(); int result = f.Result;
- 36. Coordination Data Structures (1 of 3) Block if full Concurrent Collections P C • BlockingCollection<T> P C • ConcurrentBag<T> P C • ConcurrentDictionary<TKey,TValu e> Block if empty • ConcurrentLinkedList<T> • ConcurrentQueue<T> • ConcurrentStack<T> • IProducerConsumerCollection<T> • Partitioner, Partitioner<T>, OrderablePartitioner<T>
- 37. Coordination Data Structures (2 of 3) Synchronization Primitives • Barrier • CountdownEvent Loop • ManualResetEventSlim Barrier postPhaseAction • SemaphoreSlim • SpinLock • SpinWait CountdownEvent.
- 38. Coordination Data Structures (3 of 3) Initialization Primitives • Lazy<T>, LazyVariable<T>, LazyInitializer Cancellation MyMethod( ) • ThreadLocal<T> Source Foo(…, CancellationToken ct) Cancellation Primitives Thread Boundary • CancellationToken • CancellationTokenSource Bar(…, CancellationToken ct) • ICancelableOperation ManualResetEventSlim.Wait( ct ) Cancellation Token