Shrinking the Planet—How Dedicated Optical Networks are Transforming Computational Science and Collaboration
1 like1,135 views
This document summarizes Dr. Larry Smarr's presentation on how dedicated optical networks are transforming computational science and collaboration. Some key points: 1) Dedicated optical fiber channels providing 10Gbps connections are enabling new high-performance cyberinfrastructure (CI) through optical networks like Internet2 and the OptIPuter. 2) The OptIPuter creates a planetary-scale CI through optical networking infrastructure connecting "OptIPortals" which provide visualization, data storage, and computational resources. 3) Large-scale scientific projects are generating enormous amounts of data requiring dedicated optical networks for transport, with some examples generating over 1 terabit per second of data.
Shrinking the Planet—How Dedicated Optical Networks are Transforming Computational Science and Collaboration
- 1. Shrinking the Planet—How Dedicated Optical Networks are Transforming Computational Science and Collaboration Invited Presentation to the Advisory Committee on Cyberinfrastructure National Science Foundation Arlington, VA December 16, 2008 Dr. Larry Smarr Director, California Institute for Telecommunications and Information Technology Harry E. Gruber Professor, Dept. of Computer Science and Engineering Jacobs School of Engineering, UCSD
- 2. The Three Underpinnings of Cyberinfrastructure The More Things Change, the More They Stay the Same! Data John Kogut Simulating Generator Quantum Chromodynamics Data • He Uses NCSA Telnet on the Mac Portal • The Mac Communicates with Cray Data • The Simulation Runs on the Transmission Remote Cray Source: Larry Smarr 1985
- 3. The OptIPuter Creates an Planetary Scale CI: Enabling Collaborative Data-Intensive e-Research www.evl.uic.edu/cavern/sage “OptIPlanet: The OptIPuter Global Collaboratory” – Special Section of Future Generations Computer Systems, October 1, 2008 Volume 25, Issue 2, End of Six Year ITR February 2009 Calit2 (UCSD, UCI), SDSC, and UIC Leads—Larry Smarr PI Univ. Partners: NCSA, USC, SDSU, NW, TA&M, UvA, SARA, KISTI, AIST Industry: IBM, Sun, Telcordia, Chiaro, Calient, Glimmerglass, Lucent
- 4. Dedicated Optical Fiber Channels Makes High Performance Cyberinfrastructure Possible (WDM) c=λ* f WDM Enables 10Gbps Shared Internet on One Lambda and a Personal 10Gbps Lambda on the Same Fiber!
- 5. Dedicated 10Gbps Lightpaths Tie Together State and Regional Fiber Infrastructure Interconnects Two Dozen State and Regional Internet2 Dynamic Optical Networks Circuit Network Is Now Available NLR 40 x 10Gb Wavelengths Expanding with Darkstrand to 80
- 6. Global Lambda Integrated Facility 1 to 10G Dedicated Lambda Infrastructure NSF Funds IRNC Interconnects Global OptIPortals at Public Research Innovation Centers Source: Maxine Brown, UIC and Robert Patterson, NCSA
- 7. Data Portals: From User Analysis on PCs to OptIPortals
- 8. NSF: “Distribution of Balanced Computational Capacity at the Local Nodes is as Important as the Provision of Maximum Capability of the Principal Node” • The Local Nodes Make Possible Modes of Operation and Scientific Investigations Not Possible via Remote Acess to the Central Node • A Local Node Also Provides for Much Higher I/O Rates to the User than can be Provided with Remote Access to a Central Facility • Any Commitment to Increased Computational Resources Must be Accompanied by an Expansion of Graphics Facilities to Help Manage the Increased Data Flow www.nr.com/whp/NSFCompPhys1981.pdf
- 9. Challenge—How to Bring Scalable Visualization Capability to the Supercomputer End User? 1997 1999 1999 2004 NCSA 4 MPixel LLNL 20 Mpixel Wall ORNL 35Mpixel EVEREST NSF Alliance PowerWall 2008 2004 2005 EVL 100 Mpixel LambdaVision Calit2@UCI 200 Mpixel HiPerWall NSF MRI NSF MRI TACC 307 Mpixel Stallion NSF TeraGrid A Decade of NSF Investment Two Orders of Magnitude Growth!
- 10. My OptIPortalTM – Affordable Termination Device for the OptIPuter Global Backplane • 20 Dual CPU Nodes, 20 24” Monitors, ~$50,000 • 1/4 Teraflop, 5 Terabyte Storage, 45 Mega Pixels--Nice PC! • Scalable Adaptive Graphics Environment ( SAGE) Jason Leigh, EVL-UIC Source: Phil Papadopoulos SDSC, Calit2
- 11. Rocks Cluster Toolkit is the OptIPortal Foundation and its Software Deployment System • Enables Scientists in Their Lab to Easily Reap the Benefits of Cluster Computing • Rocks is Extensible through Rolls to Build Reproducible Specialized Computing Facilities – OptIPortal Windowing and Graphics – Virtual Machines for Cluster Extension • Large User Community – Volunteer Registry of Installed Clusters > 1100 Clusters, 100K CPUS 600+TF – Active User Support List of 1800+ Users – Used at Universities Throughout the US. – Impacts NSF, NIH, DoE Funded Research • Supports RedHat Enterprise Linux et al (CentOS, Scientific Linux) – Alpha Support for Solaris Funded NSF SDCI Program award #OCI-0721623 Sept 2007
- 12. Use of OptIPortal to Interactively View Multi-Scale Biomedical Imaging 200 Megapixels! Source: Mark Ellisman, David Green: Purkinje Cells Lee, Red: Glial Cells Jason Light Blue: Nuclear DNA Leigh Two-Photon Laser Confocal Microscope Montage of 40x36=1440 Images in 3 Channels of a Mid-Sagittal Section of Rat Cerebellum Acquired Over an 8-hour Period
- 13. Scalable Displays Allow Both Global Content and Fine Detail Source: Mark Ellisman, David Lee, Jason Leigh
- 14. Allows for Interactive Zooming from Cerebellum to Individual Neurons Source: Mark Ellisman, David Lee, Jason Leigh
- 15. OptIPortals Scale to 1/3 Billion Pixels Enabling Viewing of Very Large Images or Many Simultaneous Images Spitzer Space Telescope (Infrared) NASA Earth Satellite Images Bushfires October 2007 San Diego Source: Falko Kuester, Calit2@UCSD
- 16. On-Line Resources Help You Build Your Own OptIPortal www.optiputer.net http://wiki.optiputer.net/optiportal www.evl.uic.edu/cavern/sage http://vis.ucsd.edu/~cglx/ How to Support These Global CI Communities?
- 17. Students Learn Case Studies Using Next Generation Knowledge Interface UIC Anatomy Class Training Future Workforce Using Researcher’s Tools electronic visualization laboratory, university of illinois at chicago
- 18. Disseminating Research Results at Conferences Using OptIPortals AGU 2005 AAAS 2008 Supercomputing 2008
- 19. Calit2 3D Immersive StarCAVE OptIPortal: Enables Exploration of High Resolution Simulations Connected at 50 Gb/s to Quartzite 15 Meyer Sound Speakers + Subwoofer 30 HD Projectors! Passive Polarization-- Optimized the Polarization Separation and Minimized Attenuation Source: Tom DeFanti, Greg Dawe, Calit2 Cluster with 30 Nvidia 5600 cards-60 GB Texture Memory
- 20. Cyberinfrastructure Integration: Integration of Data Generators, Transmission, and Portals
- 21. OptIPuter Software Architecture--a Service-Oriented Architecture Integrating Lambdas Into the Grid Distributed Applications/ Web Services Visualization Telescience SAGE JuxtaView Data Services LambdaRAM Vol-a-Tile Distributed Virtual Computer (DVC) API DVC Configuration DVC Runtime Library DVC DVC Services DVC Job Scheduling Communication DVC Core Services Resource Namespace Security High Speed Storage Identify/Acquire Management Management Communication Services Globus PIN/PDC GRAM GSI XIO RobuStore Discovery and Control GTP XCP UDT IP Lambdas CEP LambdaStream RBUDP
- 22. An OptIPuter Worked Example From The New Science of Metagenomics “The emerging field NRC Report: of metagenomics, where the DNA of entire Metagenomic communities of microbes data should is studied simultaneously, be made presents the greatest opportunity publicly -- perhaps since the invention of available in the microscope – international to revolutionize understanding of archives as the microbial world.” – rapidly as possible. National Research Council March 27, 2007
- 23. Calit2 is Creating a Global CI to Support Marine Microbial Metagenomics Source: CAMERA CTO Mark Ellisman Community Cyberinfrastructure for Advanced Marine Microbial Ecology Research and Analysis
- 24. Calit2 Microbial Metagenomics Cluster- Next Generation Optically Linked Science Data Server Source: Phil Papadopoulos, SDSC, Calit2 512 Processors ~200TB ~5 Teraflops Sun 1GbE X4500 ~ 200 Terabytes Storage and Storage 10GbE Switched 10GbE / Routed Core
- 25. CAMERA’s Data Center Global Microbial Metagenomics CyberCommunity Nearly 2500 Registered Users From 55 Countries
- 26. OptIPuter Persistent Infrastructure Enables Calit2 and U Washington CAMERA Collaboratory Photo Credit: Alan Decker Feb. 29, 2008 Ginger Armbrust’s Diatoms: Micrographs, Chromosomes, Genetic Assembly iHDTV: 1500 Mbits/sec Calit2 to UW Research Channel Over NLR/CENIC/PW
- 27. The Calit2 OptIPortals at UCSD and UCI Are Now a Gbit/s HD Collaboratory NASA Ames Visit Feb. 29, 2008 Calit2@ UCI wall Calit2@ UCSD wall UCSD cluster: 15 x Quad core Dell XPS with Dual nVIDIA 5600s UCI cluster: 25 x Dual Core Apple G5
- 28. U Michigan Virtual Space Interaction Testbed (VISIT) Instrumenting OptIPortals for Social Science Research • Using Cameras Embedded in the Seams of Tiled Displays and Computer Vision Techniques, we can Understand how People Interact with OptIPortals – Classify Attention, Expression, Gaze – Initial Implementation Based on Attention Interaction Design Toolkit (J. Lee, MIT) • Close to Producing Usable Eye/Nose Tracking Data using OpenCV Leading U.S. Researchers on the Social Aspects of Collaboration Source: Erik Hofer, UMich, School of Information
- 29. Green Initiative: Can Optical Fiber Replace Airline Travel for Continuing Collaborations ? Source: Maxine Brown, OptIPuter Project Manager
- 30. North American OptIPuter Team OptIPortals Calit2@UCSD Calit2@UCSD Calit2@UCSD SIO-UCSD Calit2@UCSD NCMIR-UCSD Calit2@UCSD VISIT-U Michigan EVL@UIC USGS EVL@UIC Calit2@UCI iCAIR- Northwestern U Texas A&M EVL@UIC
- 31. North American OptIPortals AOSS-U Michigan Adler Planetarium - TRCC, Chicago U Washington Chicago SDSC - UCSD DMC-U Michigan NASA Goddard TACC TeraGrid UC Irvine UCSD
- 32. North American OptIPortals Sharp Corp. Rincon Nortel Florida International U U Hawaii MIT CICESE, Mexico NCSA
- 33. Launch of the 100 Megapixel OzIPortal Kicked Off a Rapid Build Out of Australian OptIPortals January 15, 2008 Smarr OptIPortal Road Show January 15, 2008 No Calit2 Person Physically Flew to Australia to Bring This Up! Covise, Phil Weber, Jurgen Schulze, Calit2 CGLX, Kai-Uwe Doerr , Calit2 http://www.calit2.net/newsroom/release.php?id=1421
- 34. International OptIPortals Are Being Adopted--Globally Connected by GLIF AIST-Japan Osaka U-Japan Kyoto-Japan KISTI-Korea GIST-Korea NCHC-Taiwan UZurich CNIC-China Science Space Research Innovation Center SARA- Institute-Moscow Chernogolovka, Russia Netherlands Brno-Czech Republic
- 35. EVL’s SAGE OptIPortal VisualCasting Multi-Site OptIPlanet Collaboratory CENIC CalREN-XD Workshop Sept. 15, 2008 Total Aggregate VisualCasting Bandwidth for Nov. 18, 2008 EVL-UI Chicago At Supercomputing 2008 Austin, Texas Sustained 10,000-20,000 Mbps! November, 2008 Streaming 4k SC08 Bandwidth Challenge Finalist Remote: On site: U of Michigan SARA (Amsterdam) UIC/EVL U Michigan GIST / KISTI (Korea) U of Queensland Osaka Univ. (Japan) Russian Academy of Science Masaryk Univ. (CZ) Requires 10 Gbps Lightpath to Each Site Source: Jason Leigh, Luc Renambot, EVL, UI Chicago
- 36. NSF’s PRAGMA Can Drive Science-Driven Lambda-Integrated Planetary-Scale CI Systems 27 institutions in 17 countries/regions, 24 compute sites (+ 14 in preparation) Source: Peter Arzberger, Calit2@UCSD
- 37. Large Data Generators: An Exponentially Increasing Challenge
- 38. Creating Environmental Observatories – Combining OptIPortals with DataTurbine • Streaming Data Middleware System Satisfies Common Critical Infrastructure Requirements Across NSF-Sponsored Observing Systems: – Framework for Integration of Heterogeneous Instruments with Reliable Real-Time Data Transport – Sensors /Sensor Streams Become First-Class Objects – Comprehensive Suite Of Services For Data Management, Routing, Synchronization, Monitoring, and Visualization • All-Software Solution (Java) – Open Source Software- Apache 2.0 License • Used In NSF, NASA, NOAA, DOE Projects – www.dataturbine.org – NSF Support from SDCI program (Sept 07) Source: Tony Fountain, Sameer Tilak
- 39. Streaming Underwater Video From Taiwan’s Kenting Reef to Calit2’s OptIPortal My next plan is Remote Videos to stream stable Local Images and quality underwater images to Calit2, hopefully by PRAGMA 14. -- Fang-Pang to LS Jan. 1, 2008 March 6, 2008 Plan Accomplished! March 26, 2008 UCSD: Rajvikram Singh, Sameer Tilak, Jurgen Schulze, Tony Fountain, Peter Arzberger NCHC : Ebbe Strandell, Sun-In Lin, Yao-Tsung Wang, Fang-Pang Lin
- 40. The Ocean Observatory Initiative (OOI) Cyberinfrastructure is Under Development Source: John Orcutt, Matthew Arrott, SIO/Calit2 OOI CI Executive Management Team: UCSD, WHOI, Rutgers
- 41. OOI CI is Built on NLR/I2 Optical Infrastructure Physical Network Implementation Source: John Orcutt, Matthew Arrott, SIO/Calit2
- 42. NSF Instruments Generate Data at Enormous Rates-- Requiring Photonic Interconnects “The VLA facility is now able to generate 700 Gigabits/s of astronomical data and the Extended VLA will reach 3.2 Terabits/sec by 2009.” --Dr. Steven Durand, National Radio Astronomy Observatory, e-VLBI Workshop, MIT Haystack Observatory., Sep 2006. ALMA Has a Requirement for a 120 Gbps Data Rate per Telescope
- 43. Next Great Planetary Instrument: The Square Kilometer Array Requires Dedicated Fiber www.skatelescope.org Transfers Of 1 TByte Images World-wide Will Be Needed Every Minute!
- 44. CI Bottleneck: The Campus Last Mile
- 45. How Do You Get From Your Lab to the Regional Optical Networks? “Research is being stalled by ‘information overload,’ Mr. Bement said, because data from digital instruments are piling up far faster than researchers can study. In particular, he said, campus networks need to be improved. High-speed data lines crossing the nation are the equivalent of six-lane superhighways, he said. But networks at colleges and universities are not so capable. “Those massive conduits are reduced to two-lane roads at most college and university campuses,” he said. Improving cyberinfrastructure, he said, “will transform the capabilities of campus-based scientists.” -- Arden Bement, the director of the National Science Foundation www.ctwatch.org
- 46. Creating a California Cyberinfrastructure of OptIPuter “On-Ramps” to NLR & TeraGrid Resources UC Davis UC Berkeley UC San Francisco UC Merced UC Santa Cruz Creating a Critical Mass of UC Los Angeles OptIPuter End Users on UC Santa Barbara UC Riverside a Secure LambdaGrid UC Irvine UC San Diego CENIC Workshop at Calit2 Sept 15-16, 2008
- 47. CENIC’s New “Hybrid Network” - Traditional Routed IP and the New Switched Ethernet and Optical Services ~ $14M Invested in Upgrade Now Campuses Need to Upgrade Source: Jim Dolgonas, CENIC
- 48. The “Golden Spike” UCSD Experimental Optical Core: Ready to Couple Users to CENIC L1, L2, L3 Services Quartzite Communications To 10GigE cluster Goals by 2008: Core Year 3 node interfaces >= 60 endpoints at 10 GigE CENIC L1, L2 >= 30 Packet switched Wavelength Quartzite Selective Services Corewavelengths ..... Switch >= 30 Switched >= 400 Connected endpoints Lucent To 10GigE cluster node interfaces and other switches To cluster nodes ..... Glimmerglass Approximately 0.5 Tbps ..... To cluster nodes GigE Switch with Dual 10GigE Upliks Arrive at the “Optical” Production OOO Switch To cluster nodes Center of Hybrid Campus 32 10GigE ..... Switch GigE Switch with Dual 10GigE Upliks Force10 ... To Packet Switch CalREN-HPR GigE Switch with Dual 10GigE Upliks other Research nodes Cloud GigE Funded by 10GigE NSF MRI Campus Research 4 GigE 4 pair fiber Grant Cloud Cisco 6509 Juniper T320 OptIPuter Border Router Source: Phil Papadopoulos, SDSC/Calit2 (Quartzite MRI PI, OptIPuter co-PI)
- 49. OCI Should Provide End-to-End CI for Petascale End Users-TeraGrid XD? Mike Norman, SDSC October 10, 2008 Two 64K Images From a log of gas temperature log of gas density Cosmological Simulation of Galaxy Cluster Formation
- 50. Conceptual Architecture to Physically Connect Campus Resources Using Fiber Optic Networks HPC System Cluster Condo PetaScale Data Analysis UCSD Storage Facility UC Grid Pilot Digital DNA Arrays, Collections Research Mass Spec., Manager Cluster OptIPortal Microscopes, Research Genome Instrument Sequencers N x 10Gbps Source:Phil Papadopoulos, SDSC/Calit2
- 51. Threat to CI Deployment—CSE Research Needed on How to Deploy a Green CI MRI • Computer Architecture – Rajesh Gupta/CSE • Software Architecture – Amin Vahdat & Ingolf Kruger/ CSE • CineGrid Exchange – Tom DeFanti/Calit2 • Visualization – Falko Kuster/Structural Engineering • Power and Thermal Management – Tajana Rosing/CSE • Analyzing Power Consumption Data – Jim Hollan/Cog Sci http://greenlight.calit2.net
- 52. My OCI Recommendations • Build Out a National End-to-End Balanced HiPerf “Green” CI – Lead with Petascale User Lambdas & Analysis OptIPortals – Provide CI Users Access to On-Demand WAN NLR/I2 L1/L2 – Fund NLR/I2 Campus Gateway to Users ‘vBNS Connections Program’ – Add CI to NSF Transformational Review Evaluations—Also Add “Green” – Find Ways to Co-Fund International CI Collaborations • Creating and Supporting Sustainable CI and Workforce – Harvest from CISE Successful Research Grants – Use NSF SDCI-Like Programs to Sustain the CI Software – Use NSF MRIs to Prototype Campus CI Innovations – Grad, Postdoc, Career Awards for CI Development & Usage • With Other Directorates – Prototype CI Using Global Distributed Data Sharing Communities – Encourage Common CI across MREFCs – Long Term Collaborative Computational Science for Complex Problems Source: Larry Smarr, Calit2