The Component Retrieval Language
0 likes559 views
The document discusses a Component Retrieval Language (CRL) that was designed to simplify assembling complex software frameworks composed of many individual components. The CRL provides a unified syntax for checking out and updating components from version control repositories in a tool-agnostic way. It abstracts authentication procedures and is intended to be general-purpose rather than specific to any one framework like the original GetCactus script it aims to replace.
The Component Retrieval Language
- 1. SIMPLIFYING COMPLEX SOFTWARE ASSEMBLY THE COMPONENT RETRIEVAL LANGUAGE AND IMPLEMENTATION Presenter: Eric Seidel Dept. of Computer Science Co-authors: Gabrielle Allen, Steven Brandt, Frank Löffler, and Erik Schnetter City College of New York Center for Computation & Technology eric@eseidel.org Louisiana State University Wednesday, August 4, 2010
- 2. COMPONENT FRAMEWORKS • Set of individual software modules coordinated by glue framework • Each component (module) performs a specific task and encapsulates a set of related functions data • Frameworks can range from having a few components to many • Components communicate via interfaces • Used for various purposes, HPC examples include • Cactus Framework • CCA Frameworks (e.g. Caffeine) • Domain specific frameworks (e.g. Earth System Modeling Framework) Wednesday, August 4, 2010
- 3. CACTUS • Component Framework • Over 500 unique components • Distributed around the world • Flesh • Core application • Thorns • Independent modules • Perform actual computation • High Performance Computing www.cactuscode.org • Massively parallel • Runs on high end supercomputer clusters • Supports many applications • Numerical Relativity • Quantum Gravity • Computational Fluid Dynamics Wednesday, August 4, 2010
- 4. CACTUS WORKFLOW !REPOSITORY_TYPE pserver • Managed using “Thornlists” !REPOSITORY_LOCATION cvs.cactuscode.org !REPOSITORY_NAME /cactusdevcvs • Plaintext list of thorns !REPOSITORY_USER eric9 required for a specific CactusBase/Boundary configuration CactusBase/CartGrid3D CactusBase/CoordBase • Used to checkout, update, CactusBase/IOASCII CactusBase/IOBasic build, and test the source CactusBase/IOUtil CactusBase/InitBase code CactusBase/LocalInterp Wednesday, August 4, 2010
- 5. EINSTEIN TOOLKIT • Toolkit for relativistic astrophysical simulations • Developed using Cactus • Comprised of 135 thorns • Initial Data, Evolution/ Analysis methods, Utilities • First official release 2 months ago www.einsteintoolkit.org Wednesday, August 4, 2010
- 6. MOTIVATION • Distributed Software Frameworks are hard to assemble and manage • Einstein Toolkit comprised of 135 individual components • Very tedious to manually checkout or update • Large barrier to entry for new users Wednesday, August 4, 2010
- 7. VERSION CONTROL SYSTEMS • Used to track revisions in source code • Concurrent Versions System (cvs) • Released in 1990 • Uses client-server model • Server stores full history of repository • Clients retrieve specific revision • Subversion (svn) • Released in 2000 • Successor to cvs • Also uses client-server model • Git • Released in 2005 • Uses distributed model • Everyone has copy of full history http://en.wikipedia.org/wiki/ File:Revision_controlled_project_ visualization-2010-24-02.svg Wednesday, August 4, 2010
- 8. GETCACTUS • Designed to checkout and !REPOSITORY_TYPE pserver !REPOSITORY_LOCATION cvs.cactuscode.org update Cactus thorns and flesh !REPOSITORY_NAME /cactusdevcvs !REPOSITORY_USER eric9 • Specific to Cactus Framework CactusBase/Boundary • Originally designed for CVS CactusBase/CartGrid3D CactusBase/CoordBase CactusBase/IOASCII • SVN and git added later CactusBase/IOBasic CactusBase/IOUtil • Still difficult to distribute the CactusBase/InitBase framework CactusBase/LocalInterp • Users must edit the thornlist Wednesday, August 4, 2010
- 9. COMPONENT RETRIEVAL LANGUAGE # NAME is an alphanumeric or ’.’ character and • Designed to fix problems If DOCUMENT : DIRECTIVES ; nent error with original GetCactus script DIRECTIVE : DEFINE NAME ’=’ PATH EOL | CHECKOUT ’=’ COMPONENTLIST EOL rors | CHECKOUT ’=’ EOL COMPONENTLIST EOL inclu | REPO_LOC ’=’ LOC EOL error • Provides unified, tool agnostic | AUTH_LOC ’=’ LOC EOL nent | PATH_DIRECTIVE ’=’ PATH EOL and # !REPO_PATH, !CHECKOUT, !TARGET, M syntax # !ANON_PASS, !NAME | NAME_DIRECTIVE ’=’ NAME EOL # !CRL_VERSION, !AUTH_USER, whic proc # !ANON_USER, !TYPE spec • Abstracts authentication ; load the c DIRECTIVES : DIRECTIVE procedures | DIRECTIVES DIRECTIVE load ; tion, perfo LOC : PSERVER PATH # CVS repository a dis • General-Purpose | NAME ’:’ ’/’ ’/’ PATH | NAME ’@’ NAME ’:’ PATH # Git/SVN repository # Git repository tory with ; histo • No longer specific to PATH : NAME of di | ’/’ NAME only | PATH ’/’ NAME (som Cactus ; COMPONENTLIST : PATH for e 115M | COMPONENTLIST EOL PATH ; Ge can e All o are p Figure 2: Grammar for the CRL in Bison format comp Wednesday, August 4, 2010
- 10. SAMPLE CRL FILE !DEFINE ROOT = Cactus !DEFINE ARR = $ROOT/arrangements !TARGET = $ROOT !TARGET = $ARR !TYPE = svn !TYPE = git !AUTH_URL = https://svn.cactuscode.org/flesh/trunk !URL = git://github.com/ianhinder/Kranc.git !URL = http://svn.cactuscode.org/flesh/trunk !AUTH_URL = git@github.com:ianhinder/Kranc.git !CHECKOUT = Cactus !REPO_PATH= Auxiliary/Cactus !NAME = . !CHECKOUT = KrancNumericalTools/GenericFD !TARGET = $ROOT !TYPE = svn # McLachlan, the spacetime code !URL = https://svn.cct.lsu.edu/repos/numrel/$1/trunk !TARGET = $ARR !CHECKOUT = simfactory !TYPE = git !URL = git://carpetcode.dyndns.org/McLachlan !TARGET = $ARR !AUTH_URL = carpetgit@carpetcode.dyndns.org:McLachlan !TYPE = svn !REPO_PATH= $2 !AUTH_URL = https://svn.cactuscode.org/arrangements/$1/$2/trunk !CHECKOUT = McLachlan/doc McLachlan/m McLachlan/par !URL = http://svn.cactuscode.org/arrangements/$1/$2/trunk McLachlan/ML_BSSN !CHECKOUT = McLachlan/ML_BSSN_Helper CactusArchive/ADM McLachlan/ML_BSSN_O2 CactusBase/Boundary McLachlan/ML_BSSN_O2_Helper CactusBase/CartGrid3D McLachlan/ML_BSSN_Test CactusBase/CoordBase McLachlan/ML_ADMConstraints Wednesday, August 4, 2010
- 11. GETCOMPONENTS • Designed to be very modular Einstein Toolkit Einstein Toolkit Core Einstein Toolkit • Currently supports 5 svn.partnersite.org svn.einsteintoolkit.org Cactus Flesh and CCTK version control systems and svn.cactuscode.org Tools, Parameter Files, Carpet AMR & Data git.carpetcode.org http/ftp downloads svn.einsteintoolkit.org • Very easy to add more Assemble GetComponents Simulation • Can take input as local file or URL Research Groups Group Modules Individual Modules • Manages all authentication svn.groupthorns.org git.mythorns.org Group Modules issues ftp.groupthorns.org ./GetComponents http://tinyurl.com/einsteintoolkit-2010-06 Wednesday, August 4, 2010
- 12. AUTHENTICATION Anonymous mode selected? • Authentication handled entirely by yes no VCS tools Are components Is username • GetComponents stores list of available anonymously? for URL known? authenticated repositories and no users yes no Prompt for username yes • Also tracks repositories with Use anonymous Print error and ignore specified anonymous access checkout component Verify Use known • Very secure access username • GetComponents never sees any Checkout components passwords! Wednesday, August 4, 2010
- 13. CHECKOUT VS. UPDATE SPEED 1300 Serial Checkout Parallel Checkout 975 Time (seconds) 650 325 0 Abe Frost Kraken Lincoln LoneStar Longhorn Queen Bee Ranger Spur Steele TeraGrid Resource Wednesday, August 4, 2010
- 14. GETCOMPONENTS • Generating component lists is still time-consuming and tedious • Barrier/impossible for new users • Don’t need all Einstein Toolkit modules to run a simulation • How to determine which components are needed for a particular simulation? • e.g. what is needed to model two black holes, or a coastal surge? Wednesday, August 4, 2010
- 15. COMPONENT DEPENDENCIES • Dependency tracking could allow custom built simulations • Specify one component IDScalarWaveC containing data about the simulation WaveToyC • Initial values, type of Boundary CartGrid3D simulation, etc SymBase PUGH CoordBase • Then recursively check component dependencies Wednesday, August 4, 2010
- 16. COMPONENT DEPENDENCIES -- WAVETOY EXAMPLE Interface Inheritance Function Requirement IDScalarWaveC WaveBinarySource LocalInterp LocalReduce Time Direct Thorn Dependency Shared Variable Dependency Capability Requirement WaveToyC IsoSurfacer HTTPDEextra IOBasic IOAscii IOJpeg Boundary CartGrid3D HTTPD IOUtil PUGHReduce PUGHSlab jpeg6b SymBase CoordBase Socket PUGH Wednesday, August 4, 2010
- 17. COMPONENT DEPENDENCIES -- QUANTUM GRAVITY Interface Inheritance Function Requirement Nerve PUGH Direct Thorn Dependency Shared Variable Dependency Capability Requirement AntichainEvol RandomAntichain CFlatSprinkle BinaryCauset MonteCarlo Distributions IOUtil RNGs Wednesday, August 4, 2010
- 18. COMPONENT DEPENDENCIES -- EINSTEIN TOOLKIT ellbase idbrilldata admmacros idfileadm httpd time idaxioddbrillbh terminationtrigger httpdextra socket localreduce ml_bssn_test exact Interface Inheritance aeilocalinterp coordgauge Function Requirement fortran Direct Thorn Dependency idaxibrillbh Shared Variable Dependency tatelliptic idconstraintviolate idlinearwaves Capability Requirement adm formaline iobasic ioutil ml_bssn_helper ellsor iojpeg mol localinterp distortedbhivp idanalyticbh carpetevolutionmask newrad tmunubase admanalysis calck ahfinder ehfinder iohdf5util extract carpetiobasic ml_bssn_o2_helper rotatingdbhivp staticconformal timerreport admconstraints tgrtensor legoexcision weylscal4 quasilocalmeasures ahfinderdirect cartgrid3d admbase ml_bssn carpetioscalar carpetioascii boundary carpetslab dissipation multipole carpetmask noexcision slabtest ml_bssn_o2 carpet genericfd ml_admconstraints twopunctures rotatingsymmetry90 iohdf5 noise spacemask ioascii grhydro_initdata carpetlib ml_admquantities periodic pugh grhydro coordbase sphericalsurface pughslab meudon_bin_bh tovsolver rotatingsymmetry180 carpetinterp reflectionsymmetry carpetregrid carpetiohdf5 cartoon2d hydro_initexcision symbase meudon_mag_ns hydro_analysis meudon_bin_ns slab jpeg6b nice norms eosg_idealfluid lapack pughreduce admcoupling eos_polytrope carpetinterp2 carpetregrid2 eos_base initbase gsl carpetreduce loopcontrol hydrobase summationbyparts eos_idealfluid hdf5 blas nanchecker pughinterp carpettracker eosg_polytrope eos_hybrid eosg_base constants setmask_sphericalsurface lorene eosg_hybrid Wednesday, August 4, 2010
- 19. COMPONENT DEPENDENCIES -- EINSTEIN TOOLKIT ellbase idbrilldata admmacros idfileadm httpd idaxioddbrillbh time terminationtrigger httpdextra socket localreduce ml_bssn_test exact Interface Inheritance aeilocalinterp coordgauge Function Requirement fortran Direct Thorn Dependency idaxibrillbh Shared Variable Dependency tatelliptic idconstraintviolate idlinearwaves Capability Requirement adm formaline iobasic ioutil ml_bssn_helper ellsor iojpeg mol localinterp distortedbhivp idanalyticbh carpetevolutionmask newrad tmunubase admanalysis calck ahfinder ehfinder iohdf5util extract carpetiobasic ml_bssn_o2_helper rotatingdbhivp staticconformal timerreport admconstraints tgrtensor legoexcision weylscal4 quasilocalmeasures ahfinderdirect cartgrid3d admbase ml_bssn carpetioscalar carpetioascii boundary carpetslab dissipation multipole carpetmask noexcision slabtest ml_bssn_o2 carpet genericfd ml_admconstraints twopunctures rotatingsymmetry90 iohdf5 noise spacemask ioascii grhydro_initdata carpetlib ml_admquantities periodic pugh grhydro coordbase sphericalsurface pughslab meudon_bin_bh tovsolver rotatingsymmetry180 carpetinterp reflectionsymmetry carpetregrid carpetiohdf5 cartoon2d hydro_initexcision symbase meudon_mag_ns hydro_analysis meudon_bin_ns slab jpeg6b nice norms eosg_idealfluid lapack pughreduce admcoupling eos_polytrope carpetinterp2 carpetregrid2 eos_base initbase gsl carpetreduce loopcontrol hydrobase summationbyparts eos_idealfluid hdf5 pughinterp eosg_polytrope blas nanchecker carpettracker eos_hybrid eosg_base constants setmask_sphericalsurface lorene eosg_hybrid Wednesday, August 4, 2010
- 20. DISTRIBUTION • GetComponents is freely available with an open-source license • www.eseidel.org/download/GetComponents • Full documentation available • ./GetComponents --man Wednesday, August 4, 2010
- 21. ACKNOWLEDGEMENTS • Many thanks to Gabrielle Allen, Steve Brandt, Frank Löffler, and Erik Schnetter Wednesday, August 4, 2010