Modeling and Simulation of Parallel and Distributed Computing Systems with SimGrid, WRENCH, and WfCommons
0 likes340 views
The document discusses the challenges in experimental research within parallel and distributed computing systems and proposes the use of simulation as a solution. It highlights the capabilities of simulation frameworks like SimGrid, Wrench, and WFCommons in providing accurate and scalable simulations for workflow management and CI systems. The frameworks facilitate a simulation-driven engineering lifecycle, allowing for effective testing and evaluation of complex workflows in scientific research.
![https://wrench-project.org https://wfcommons.org https://simgrid.org
SimGrid’s
Philosophy
7
SimGrid provides
low-level abstractions
you can do anything with it
implementing a simulation of a complex system
is cumbersome
Critical analysis: In [Kecskemeti et al.’14] pinpoints exactly the above trade-off:
Summary: SimGrid is more scalable and validated than competing frameworks,
but just too much work when wanting to simulate a complex CI system](https://image.slidesharecdn.com/simgrid-wrench-wfcommons-210827184945/85/Modeling-and-Simulation-of-Parallel-and-Distributed-Computing-Systems-with-SimGrid-WRENCH-and-WfCommons-7-320.jpg)
Modeling and Simulation of Parallel and Distributed Computing Systems with SimGrid, WRENCH, and WfCommons
- 1. Modeling and Simulation of Parallel and Distributed Computing Systems with SimGrid, WRENCH, and WfCommons Rafael Ferreira da Silva, Ph.D. https://rafaelsilva.com
- 2. https://wrench-project.org https://wfcommons.org https://simgrid.org disconnect between CI Theory and Practice 2 Many theoretical results are not useful to practitioners One well-known reason is that theoretical results are obtained with models that, to be tractable, are often unrealistic or unattainable in practice As a result, practical work must be experimental
- 3. https://wrench-project.org https://wfcommons.org https://simgrid.org real-world Experiments are Limited 3 One is limited to particular platform configurations (and sub-configurations) How can “what if?” scenarios be explored? How can generality be claimed? One is limited by specifics of the software infrastructure that impose constraints on CI application executions Modifying complex software stacks (often written by others) just to test out ideas is not feasible Limited Experimental Scope impedes progress / discovery
- 4. https://wrench-project.org https://wfcommons.org https://simgrid.org resorting to Simulation 4 When one works in an experimental field in which experiments are problematic, one resorts to simulation In some fields of Computer Science simulation is a standard research and development methodology So, what about using simulation to drive CI Research & Development?
- 5. https://wrench-project.org https://wfcommons.org https://simgrid.org Simulation-driven engineering life cycle 5 Research Idea Design of Research Solution Design of Workflow Simulator Accurate Workflow Simulator Experimental Simulation Evaluation of Simulation Results Research Product Implementation onto Workflow System unsatisfactory results The ability to easily develop accurate CI simulators, from which research products evaluated via experimental simulation could be seamlessly integrated into actual CI platforms R. Ferreira da Silva, H. Casanova, R. Tanaka, F. Suter (2019). Bridging Concepts and Practice in eScience via Simulation-driven Engineering. Workshop on Bridging from Concepts to Data and Computation for EScience (BC2DC’19), 15th International Conference on EScience (EScience). DOI: 10.1109/eScience.2019.00084
- 6. https://wrench-project.org https://wfcommons.org https://simgrid.org The SimGrid simulation framework 6 H. Casanova, A. Giersch, A. Legrand, M. Quinson, F. Suter. Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms. Journal of Parallel and Distributed Computing, Elsevier, 2014, 74 (10), pp.2899-2917. http://hal.inria.fr/hal-01017319 SimGrid is a mature research project Provides simulation models of hardware/software stacks that are accurate (validated/invalidated) and scalable (low computational complexity, low memory footprint) SimGrid is open source usable software Vibrant developer and user community, continuous integration testing, one release every 6 months SimGrid is versatile Used for (combinations of) Grid, HPC, Peer-to-Peer, Cloud, and Fog simulation projects https://simgrid.org First developed in 2000 latest release: v3.28 (July 2021)
- 7. https://wrench-project.org https://wfcommons.org https://simgrid.org SimGrid’s Philosophy 7 SimGrid provides low-level abstractions you can do anything with it implementing a simulation of a complex system is cumbersome Critical analysis: In [Kecskemeti et al.’14] pinpoints exactly the above trade-off: Summary: SimGrid is more scalable and validated than competing frameworks, but just too much work when wanting to simulate a complex CI system
- 9. https://wrench-project.org https://wfcommons.org https://simgrid.org SimGrid’s Key Concepts 9 S4U interface (SimGrid for you) Actor Engine Elements Mailbox Active entities executing an application Simulation engine (singleton) Communication rendezvous, with which actors meet each other Disk Host Resources Link Resource on which actors can write and read data Actor location, providing computational power Interconnecting hosts https://simgrid.org/doc/latest/app_s4u.html
- 10. https://wrench-project.org https://wfcommons.org https://simgrid.org The WRENCH simulation framework 10 H. Casanova, R. Ferreira da Silva, R. Tanaka, S. Pandey, G. Jethwani, W. Koch, S. Albrecht, J. Oeth, and F. Suter, "Developing Accurate and Scalable Simulators of Production Workflow Management Systems with WRENCH", Future Generation Computer Systems, vol. 112, 2020. DOI: 10.1016/j.future.2020.05.030 https://wrench-project.org First developed in 2017 latest release: v1.9 (August 2021) simulation building blocks Reuse of existing and implementation of novel simulated core CI services simulation integration Development of software elements that make it possible to augment existing systems with simulation capabilities in a language-agnostic manner Workflow Systems simulation accuracy Reliance on existing and development of novel accurate and validated simulation models simulation scalability Ability to run large simulations quickly on a single computer with low compute, memory, and energy footprints Developed software elements make it possible to augment CI systems with simulation capabilities to transform the way in which these systems are experimentally evaluated The above will be demonstrated for several production Workflow Systems that support critical scientific applications
- 11. https://wrench-project.org https://wfcommons.org https://simgrid.org 11 WRENCH’s Simulated Core CI Services Compute Services Provide mechanisms for executing application tasks, which entail I/O and computation cloud bare-metal virtualized cluster batch-scheduled cluster Storage Services Store application files, reading/writing by the compute services File Registry Services Databases of key-value pairs of storage services and files replicas Network Proximity Services Monitor the network and provide host-to-host network distances Workflow Systems Mechanisms for executing workflow applications disk object storage I/O cache burst buffer scheduling energy resource provisioning Simulation core All necessary simulation models and base abstractions (computing, communicating, storing), provided by SimGrid
- 12. https://wrench-project.org https://wfcommons.org https://simgrid.org 12 WRENCH’s Simulation Accuracy and Scalability Accuracy The ability to capture the behavior of a real-world system with as little bias as possible Scalability The ability to simulate large systems with as few CPU cycles and bytes of RAM as possible
- 13. https://wrench-project.org https://wfcommons.org https://simgrid.org The WfCommons workflow research and development framework 13 T. Coleman, H. Casanova, L. Pottier, M. Kaushik, E. Deelman, and R. Ferreira da Silva, "WfCommons: A Framework for Enabling Scientific Workflow Research and Development," arXiv preprint arXiv:2105.14352, 2021. https://wfcommons.org First developed in 2020 latest release: v0.7 (August 2021) A traditional approach for testing, evaluating, and evolving workflow systems is to use full-fledged software stacks to execute applications on distributed platforms and testbeds WfCommons is a framework that provides a collection of tools for analyzing workflow execution traces, producing realistic synthetic workflow traces, and simulating workflow executions
- 14. https://wrench-project.org https://wfcommons.org https://simgrid.org 14 WfCommons’ Concept WfFormat WfInstances WfChef WfGen WfSim Catalogs of workflow instances Common JSON workflow instance format Automates the construction of synthetic workflow generators Generation of realistic synthetic workflow instances Catalogs of workflow simulators supporting WfFormat
- 15. https://wrench-project.org https://wfcommons.org https://simgrid.org SimGrid, WRENCH, and WfCommons in Numbers 15 since 2020 since 2018 since 2001 165 workflow execution instances 9 workflow applications 3 workflow simulators 4 WfCommons-enabled research publications 6 stable software releases 13 stable software releases 30+ stable software releases 18 WRENCH-enabled research publications 540+ SimGrid-enabled research publications 6 open source software leveraging SimGrid 13 simulators for pedagogic modules 40+ contributors for SimGrid’s core software 16 contributors for WRENCH’s core software
- 16. Modeling and Simulation of Parallel and Distributed Computing Systems with SimGrid, WRENCH, and WfCommons Rafael Ferreira da Silva, Ph.D. https://rafaelsilva.com 8/27/21 https://wrench-project.org https://wfcommons.org https://simgrid.org Thank You