Ecss des
Download as PPT, PDF1 like408 views
The document describes a project to generate synthetic sky catalogs for the Dark Energy Survey through large cosmological simulations. An automated workflow was developed using Apache Airavata to run the simulations across multiple XSEDE resources. The workflow involves running CAMB to generate power spectra, 2LPTic for initial conditions, and LGadget for evolution. It was able to run four full simulations using over 300,000 service units faster than manual submission. Lessons learned include differences in MPI libraries and job scripts on resources and time needed to migrate codes and data. Future goals include post-processing integration and moving to new XSEDE resources like Trestles and Stampede.
Ecss des
- 1. Efficient Production of Synthetic Skies for the Dark Energy Survey Raminder Singh Science Gateways Group Indiana University, Bloomington. ramifnu@iu.edu
- 2. Background & Explanation • First project to combine four different statistical methods of dark matter and dark energy. • Baryon acoustic oscillations in the matter power spectrum. • Abundance and spatial distribution of galaxy groups and clusters. • Weak gravitational lensing by large-scale structure. • Type Ia supernovae - are quasi-independent. • 5000-square degree survey of cosmic structure traced by galaxies • Simulation Working Group is generating simulations for galaxy yields in various cosmologies • Analysis of these simulated catalogs offers a quality assurance capability for cosmological and astrophysical analysis of upcoming DES telescope data.
- 3. Blind Cosmology Challenge (BCC) • N-Body simulation to simulate the dark matter • Post-processed using an empirical approach to link galaxy properties to dark matter structures (halos) • Gravitational lensing shear (new Spherical Harmonic Tree code) • ADDGALS methodology, empirical tuning. Figure: Processing steps to build a synthetic galaxy catalog. Xbaya workflow currently controls the top- most element (N-body simulations) which consists of methods to sample a cosmological power spectrum (ps), generating an initial set of particles (ic) and evolving the particles forward in time with Gadget (N- body). The remaining methods are run manually on distributed resources.
- 4. Project Plan • Project requested for ECSS support to automate and optimize processing on XSEDE resources. • ECSS staff, Dora Cai worked with the group on the work plan. • Work plan was discussed and a timeline were given to each task • Work plan called for porting all codes to TACC-ranger
- 5. Main Goals • Automated (re)submission of jobs to improve manual inefficiencies Walltime limits of TACC Ranger long queue is 48 hrs so jobs need restart. Multiple simulations can run in parallel • Automatic data archival Data need to be archived on Ranch Data need to moved to SLAC for post processing • Provenance • User portal
- 6. Development Roadmap XSEDE Quarterly Developments Lesson Learned July - Sept 2011 • Deployed/Tested the N-Body simulation • Added module loading code on Ranger. support on Ranger using • Developed/Tested individual codes using Gram. RSL parameters Apache Airavata. mod, mod_del • Developed BCC Parameter Maker for workflow configuration Sept - Dec 2011 • Enhanced Parameter maker script to • Queue walltime 48 hr accept inputs from Airavata Workflow. limits of Ranger • Full workflow run for Single N-Body smaller simulation Jan - Mar 2012 • Medium scale testing • Globus online client • Restart capability using Do-While library is required to contracts initiate transfers from • Running multiple simulations in parallel workflow for production runs. • Evaluated file movement using GridFTP, bbcp, Globus Online.
- 7. Development Roadmap XSEDE Quarterly Developments Learn Learned April - June2012 • Ran few more production simulations 2 • Luster issue with to in parallel many files created in a • Changed I/O routines to fix issues. single folder • Able to produce full simulation data for • Gram canceling jobs post processing after few hours in case of connection timeout with client July – Sept 2012 • Investigate the job canceling issue and • Globus team confirmed reported to Globus team that issue is not from • Changed COG API to add debug server side but client statements to debug issue • didn't give proper error • Ran more production boxes to debug • Migration to SDSC trestles • MPI library loading and RSL parameter resolved using Gram on Trestles Oct – Dec 2012 • Able to run 2 full 4 box simulations • Restart files were not using workflow. written properly • Able to resolve Job cancel issue for the because of some new latest run. exception
- 8. Xbaya Workflow • Three large-volume realizations with 20483 particles using 1024 processors • One smaller volume of 14003 particles using 512 processors. • These 4 boxes need about 300,000 SUs • Each cosmology simulation entails roughly 56TB of data.
- 9. Workflow Description • BCC Parameter Maker This initial setup code is written as a python script and prepares necessary configurations and parameter files for the workflow execution. • CAMB The CAMB (Code for Anisotropies in the Microwave Background) application computes the power spectrum of dark matter, which is necessary for generating the simulation initial conditions. • 2LPTic The Second-order Lagrangian Perturbation Theory initial conditions code (2LPTic) is programmed using Message Passing (MPI) C code that computes the initial conditions for the simulation from parameters and an input power spectrum generated by CAMB. • LGadget The LGadget simulation code is MPI based C code that uses a TreePM algorithm to evolve a gravitational N-body system. The outputs of this step are system state snapshot files, as well as lightcone files, and some properties of the matter distribution, including diagnostics such as total system energies and momenta. The total output from LGadget depends on resolution and the number of system snapshots stored, and approaches close to 10 TeraBytes for large DES simulation volumes.
- 10. Production results • We have run four full size boxes with Airavata. The following table compares how time improvement is used by Airavata compared to manually submitting jobs. * ignores 1.5 day delay due to network connection error with job listener and subsequent gram5 job deletion
- 11. Lessons learned • MPI libraries and job scripts can be different on different resources. User needs to experiment to learn. • User scratch policies can be different on different machines so it can talk some time to migrate between resources. • Migration of working codes from one machine to another can take weeks to months • Grid Services and client libraries need 1st class support XSEDE ticketing system is your Best Friend!
- 12. Future Goals • Migrate to SDSC Trestles for next production run. • Group is planning to work with Apache Airavata for future extensions. • Implement intermediate start of workflow in case of failure based on Provenance information. • Post processing • Plan for TACC Stampede migration. • Currently we are using Globus Online GUI interface for file transfer but would like to integrate using API’s with the Workflow • Migrate post processing and quality assurance code on XSEDE and develop post processing workflow • Try to integrate Post processing steps on SLAC
- 13. Team & Publication DES Simulation Working group August E. Evrard, Departments of Physics and Astronomy(Michigan) Brandon Erickson, grad student (Michigan) Risa Wechsler, asst. professor (Stanford/SLAC) Michael Busha, postdoc (Zurich) Matt Becker, grad student (Chicago) Andrey V. Kravtsov, Department of Astronomy and Astrophysics (Chicago) ECSS Suresh Marru, Science Gateways Group(Indiana) Marlon Pierce Science Gateways Group(Indiana) Lars Koesterke, TACC (Texas) Dora Cai, NCSA (Illinois) Publication: XSDEDE 12: Brandon M. S. Erickson, Raminderjeet Singh, August E. Evrard, Matthew R. Becker, Michael T. Busha, Andrey V. Kravtsov, Suresh Marru, Marlon Pierce, and Risa H. Wechsler. 2012. A high throughput workflow environment for cosmological simulations. In Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond (XSEDE '12). ACM, New York, NY, USA, , Article 34 , 8 pages. DOI=10.1145/2335755.2335830 http://doi.acm.org/10.1145/2335755.2335830
Editor's Notes
- #3: Dark energy survey is the first project to combine four different statistical methods of dark matter and dark energy. It is a 5000 square degree survey of cosmic structures traced by galaxies. Simulation working group is focused on generating synthetic skies for dark energy survey. This synthetic data is to provide quality assurance for the astronomy pipelines and the galaxy catalogs.