![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-2-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-4-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-5-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-6-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-7-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-8-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-10-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-16-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-17-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-21-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-22-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-23-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-24-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-25-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-32-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-38-320.jpg)
![Performance Parity Between
Linux and Windows
1 Million active Cells, 1000 wells, Blackoil
5500
5000
Elapsed Time [secs]
4500
4000
3500
3000
2500
2000
1500
1000
500
Cores 1 2 4 6 8 16 24 32 48
RedHat 5 U3 5200.43 3385.17 3095.72 2281.25 1790.59 1014.42 776.71 638.43 621.42
Win HPC R2 SP1 5404.38 3298.55 3175.9 2171.37 1736.11 992.82 745.43 610.88 549.74
Make your choice based on features and TCO…](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-40-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-48-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-49-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-51-320.jpg)
![[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)](https://image.slidesharecdn.com/drich110413cs264-110428210736-phpapp01/85/Harvard-CS264-15a-The-Onset-of-Parallelism-Changes-in-Computer-Architecture-and-Microsoft-s-Role-in-the-Transition-David-Rich-Microsoft-Research-52-320.jpg)
[Harvard CS264] 15a - The Onset of Parallelism, Changes in Computer Architecture and Microsoft's Role in the Transition (David Rich, Microsoft Research)
- 1. David Rich April 2011 The Onset of Parallelism Changes in computer architecture and Microsoft’s role in the transition
- 3. Your introduction – some questions… ! What kind of software do you see yourself working on in the future? Scientific? Web? Games? Business? ! Have you worked on a distributed app? MPI? ! Have you used Visual Studio? ! Which will limit performance in the future: Power consumption? Latency? Lack of parallelism? Bugs?
- 9. ! Made in 1922 by Robert Flaherty ! Considered to be the first full length documentary -though some scenes were staged ! http://en.wikipedia.org/ wiki/ Nanook_of_the_north
- 11. Job Specialization Bricklayer / Masonry Industrial Pipefitter Carpenter (construction) Caulker / Pointer / Cleaners Industrial Welder Cement Mason (construction) Construction Lineman Ironworker, Structural Drywall Finisher/Taper Laborer Marble Setter, Masonry Electrician, Elevator Mechanic Millwright Construction Electrician, HVAC-- Environmental Control System Machinery Erector Servicer & Installer Operating Engineer Electrician, General Painter--Decorator / Traffic Journeyman (Inside) Control Painter Electrician, Limited Energy Pile Driver Technician A Pipefitter • What about? Electrician, Limited Energy Plasterer Technician B Plumber – Architect Electrician, Limited Renewable Renewable Energy Technician Energy Technician – Surveyor Roofer Electrician, Limited Residential Scaffold Erector – Inspector Electrician, Sign Maker- Sheet Metal Worker Erector / Sign Hanger / Sign Solar Heating/Cooling • Or people that work in Assembler-Fabricator Exterior/Interior Specialist Systems Installer the companies that Sprinkler Fitter (metal framing & drywall) Steamfitter produce pre-fab Finisher, Masonry Floorcoverer Technical Engineer components? Terrazzo Worker, Masonry Glazier (construction) – Pipes, wires, windows, Heat / Frost Insulator Tilesetter, Masonry Tree Trimmer, Power Line fixtures, etc. Heavy Duty Repairer Truck Driver (Heavy)
- 12. Guggenheim Museum in Bilbao Acorn pre-fab house
- 13. Preparing for the Future – What Will Your Machine Look Like in 5 to 10 Years? ! Look at the Top500, predict and divide: 1. At any point in time, most organizations can afford a machine which is 1/1000th the size of the #1 machine on the Top500 2. Exaflop comes from 2x efficiency, 2x frequency and 100x the cores Today’s #1 Test: Is this within Exaflop Your Future Tianhe-1A your budget? Platform (1/1000th) Perf: 2.5 PFs 250 TFs 1000PFs 1PF Nodes 7,168 7 500,000? 500? Cores X86: 86,016 X86: 86 -- ~14 Xeons 130 Million 130 GPU: GPU: 3,211 -- ~7 Tesla Thousand 3,211,164 Cores…
- 14. Core Counts On the Rise 3,500,000 Number of Cores in Top500 #1 Over Time Tianhe-1A GPUs Get to #1... 3,000,000 250,000 Jaguar 2,500,000200,000 2,000,000 150,000 Blue Cores Gene 1,500,000 RoadRunner 100,000 1,000,000 50,000 500,000 ASCI Earth ASCI Red White Simulator Fujitsu - - Jun 93 Nov 93 Jun 94 Nov 94 Jun 95 Nov 95 Jun 96 Nov 96 Jun 97 Nov 97 Jun 98 Nov 98 Jun 99 Nov 99 Jun 00 Nov 00 Jun 01 Nov 01 Jun 02 Nov 02 Jun 03 Nov 03 Jun 05 Jun 05 Nov 05 Jun 06 Nov 06 Jun 07 Nov 07 Jun 08 Nov 08 June 09 Nov 09 Jun 10 Nov 10 14
- 15. Good News: Everybody gets a Petaflop! Bad News: You have to find 200,000 way parallelism
- 18. Caveat: No biology since high school…
- 19. Niche vs. Commodity Computing in HPC “Perfect Predator” Homogeneity Performance growth with decreasing cost and no code Commodity changes. Clusters ? Horizontal Industry 64bit x86 + Linux Cluster of SMP IBM, Dell–many HP, Commodity Clusters RISC + *nix + others Plus: MPI GPU, Multicore, Cloud, IBM, Digital, FPGA, “big data” & Vertically Integrated SGI… Windows! Single Machines IBM, Digital, Cray, HP (Apollo, Data General, ? Prime, Masscomp, Gould…) 80’s 90’s 00’s 10’s 20’s
- 20. www.calxeda.com
- 26. 2 years 6 years 12 MM users 2 Bil emails/day 7 years 5 Bil conf mins/yr. 11 years Update 12 Bil queries/mo. 12 years 40 Petabytes/ mo. 13 years 500 Million active Windows Live IDs 550 MM users/ 9.9 Billion messages / day via WL Messenger mo. Over 1 Million BPOS Users in 36 Countries 15 years 450 MM users
- 27. Microsoft’s Datacenter Evolution Datacenter Co- Quincy and San Chicago and Dublin Modular Datacenter Location Antonio Generation 3 Generation 4 Generation 1 Generation 2 Facility PAC Server Capacity Time to Market Lower TCO
- 28. Generation 3 - Chicago Data Center $500M+ investment 1.5 million person hours-of-labor 3000 construction related jobs 3400 tons of steel 707,000 sq ft 190 miles of conduit 2400 tons of copper 7.5 miles of chilled water piping 26,000 cubic yards of concrete
- 29. Visual Studio ! Visual Studio is used by over half of the professional programmers in the world ! VS2010 – released a year ago – has been downloaded over 7 million times (more than 4 million extension downloads) ! Main point: when we release a new capability into Visual Studio it automatically gets large adoption ! (story about the ISC developers)
- 30. Microsoft and GPUs The volume business….
- 31. GPU Hardware Evolution Year Version Defining Feature 1996 DirectX3 Hardware rasterization 1997 DirectX5 2 Shading options to select 1998 DirectX6 Multi-texture operations 1999 DirectX7 Vertex Processing in hardware 2000 DirectX8 Programmable Shaders: Vertex and Pixel 2002 DirectX9 High Level Shading Language, 32 instr 2003 DirectX9c 1000s of instructions per shader 2006 DirectX10 Unified Shaders: consistent shader models 2009 DirectX11 Compute Shader: explicit SIMD, random I/O
- 33. The GPGPU Software Stack High level tools and ! Windows has broad libraries support at all levels: PGI “x86 CUDA”, CAPS, Culatools, Volara, • Supports all HW Acceleware • Each of CUDA, OpenCL and Low Level Programming DirectCompute CUDA, OpenCL, DirectCompute • Almost all high level tools and libraries Hardware GPU: AMD & NVIDIA Mullticore x86: AMD & Intel
- 34. DirectCompute ! What is DirectCompute? • Microso3’s GPGPU Programming Solu<on • API of the DirectX Family • Component of the Direct3D API ! Why Use DirectCompute Over Other APIs? • Interoperability with rest of 2D, 3D, Video rendering APIs (display computed results) • Cross-‐hardware compa<bility • Feature compa<bility guarantees • Access to fixed-‐func<on hardware ! Used extensively by the gaming community http://msdn.microsoft.com/directx
- 35. GPGPU Development on Windows ! Choice: CUDA, OpenCL or DirectCompute ! Tools and libraries; Nsight and Visual Studio, PGI, CAPS, MATLAB, Jacket, PyCUDA, Quantifi, CUDA.NET, Culatools, NAG, Scicomp… many others ! NVIDIA reports that over 80% of CUDA SDK downloads are for Windows
- 36. Microsoft and NVIDIA NVIDIA’s Parallel Nsight is integrated with Microsoft’s Visual Studio
- 37. MATLAB Computer Cluster Desktop Computer MATLAB Distributed Computing Server Parallel Computing Toolbox Windows HPC Server Workers
- 39. Cluster HPC ISV / OSS Excel MPI SOA Applications Applications HPC Middleware Pack SOA HPC Edition Operating Systems On Premise Cluster Computing *Note that in SP1 support for MPI applications on Azure does not exist.
- 40. Performance Parity Between Linux and Windows 1 Million active Cells, 1000 wells, Blackoil 5500 5000 Elapsed Time [secs] 4500 4000 3500 3000 2500 2000 1500 1000 500 Cores 1 2 4 6 8 16 24 32 48 RedHat 5 U3 5200.43 3385.17 3095.72 2281.25 1790.59 1014.42 776.71 638.43 621.42 Win HPC R2 SP1 5404.38 3298.55 3175.9 2171.37 1736.11 992.82 745.43 610.88 549.74 Make your choice based on features and TCO…
- 41. NEW
- 42. ! Connects to the cluster as a SOA client Excel SOA Client ! VSTO code in workbook calls out to SOA Service ! Input and output managed by Excel developer ! Run multiple instances of Excel 2010 on an HPC Cluster Excel Workbook on ! Each instance runs an iteration of the same workbook the Cluster ! Can be launched from Excel 2010 or a Windows program NEW ! Excel Dialog Suppression ! Run User Defined Functions in parallel on a cluster ! Excel 2010 includes a new API and options for HPC Excel UDF on the cluster Cluster ! Support for .XLL files developed through Excel SDK NEW ! Easy to develop on a desktop and then deploy to a cluster
- 43. ! Use Azure servers to run HPC compute Jobs ! Can be used to “burst-out” to the cloud to handle peak demand ! Can create clusters that include dedicated on-premise servers, non-dedicated workstations and shared Azure servers ! Jobs can run unchanged across all 3 types of compute nodes (no support for MPI in SP1) ! Azure nodes are added to cluster using the Administration console (just like Workstation nodes) HPC Clients Azure Head & Broker Nodes Jobs Requests Azure Gateway
- 44. Compute Nodes On-Premise and in Azure Simultaneously HPC Head Node Desktops • “Burst” into cloud on- demand while keeping control over data and corporate policies Broker Node On-premise Compute Nodes • Pay only for what you use • A stepping stone to hybrid Azure and public clouds. • Dynamically adjust how Azure much runs on-premise and in the cloud Compute Proxies Compute Instances
- 45. Parallel Development “Combined with Intel Parallel Studio, I think it is reasonable to say that Windows has the richest and most complete set of tools for multicore programming”. -- James Reinders, Intel, 12-April-2010
- 46. Solution Begins with DEVELOPERS Make it easier to express and manage the correctness, efficiency and maintainability of parallelism on Microsoft platforms for developers of all skill levels Enable developers to Simplify the express parallelism process of easily and focus on designing and Improve the testing parallel the problem to be efficiency and solved applications scalability of parallel applications
- 47. Visual Studio 2010 Tools, Programming Models, Runtimes Tools Programming Models Parallel LINQ Parallel Parallel Agents Debugger Task Parallel Pattern Tool Library Library Library Data Structures Data Structures Windows Visual Studio .NET Framework 4 Visual C++ 10 Concurrency Runtime IDE Profiler ThreadPool Concurrenc Task Scheduler y Task Scheduler Analysis Resource Manager Resource Manager Operating UMS System Windows Threads Threads Managed Native Tooling
- 50. World’s Fastest House Construction Three and a Half Hours
- 53. http://www.microsoft.com/ hpc David Rich darich at microsoft.com
- 54. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.