A package system for maintaining large model distributions in vle software
1 like506 views
The document discusses the development of a package system for the Virtual Laboratory Environment (VLE) software, aimed at enhancing collaboration among researchers in agricultural modeling. It outlines the challenges faced in sharing and reusing models and details how the package system facilitates the integration of models, data, and libraries while improving model maintenance. The results indicate that the package system has been successfully implemented and is being actively used to aggregate models and data within the research community.
![How to develop models ?
For sub-formalisms
We provide simplified API (object oriented development)
We provide C++ generators
For example, with the difference equation sub-formalism (which inherits
the previous Dynamics class), only the compute function is available:
class MyModel : public vle :: extension :: D i f f e r e n c e E q u a t i o n
{
[...]
virtual double compute ( const Time & time )
{
x = y ( -1) + z ( -1);
y = z ( -1);
}
};
Quesnel et al. (INRA) Package system for VLE 12 / 32](https://image.slidesharecdn.com/apackagesystemformaintaininglargemodeldistributionsinvlesoftware-120807013139-phpapp02/85/A-package-system-for-maintaining-large-model-distributions-in-vle-software-12-320.jpg)
A package system for maintaining large model distributions in vle software
- 1. A Package System for Maintaining Large Model Distributions in VLE Software WETICE/Comets 2012 Gauthier Quesnel, Ronan Tr´pos e INRA - French National Institute for Agricultural Research MIA - The Division of Applied Mathematics and Computer Sciences BIA - Biometrics and Artificial Intelligence Unit Quesnel et al. (INRA) Package system for VLE 1 / 32
- 2. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 2 / 32
- 3. VLE Virtual Laboratory Environment A multi-modeling and simulation environment: A DEVS simulation engine A command line interface : VLE A modeling graphical user interface: GVLE A python package to develop web services: PyVLE A R package to analyze and visualize simulations results: RVLE VLE is developed with the support of: INRA, France Computer Science Laboratory, Calais, France CIRAD, Montpellier, France Computer Science Laboratory, University of Mons-Hainaut, Belgium Since 2007, VLE has more and more users and developers Quesnel et al. (INRA) Package system for VLE 3 / 32
- 4. RECORD, an INRA project An open platform for modeling and simulation of agro-ecosystems Project initiated in 2006 which relies on VLE software That makes multi-disciplinary work easier Capacity to take into account all the different elements of cropping systems Each element can be expressed in the most suited mathematical formalism Target: all researchers and engineers at INRA that use dynamic systems Quesnel et al. (INRA) Package system for VLE 4 / 32
- 5. Need to improve the collaboration Needs for a collaboration system due to the heterogeneity of domains involved: agronomy, economists, statistician, computer scientists, . . . Needs in sharing models; eg. integration of models at the plot scale to little territories scale (thousands of plots) Needs in distributing existing models; eg. Stics (more than 25000 lines of fortran code) Needs in helping the collaboration between modelers. More and more people trained to the platform through 3 days practice sessions (2 per year, about 200 persons trained since 2007) and access to an E-learning web site Quesnel et al. (INRA) Package system for VLE 5 / 32
- 6. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 6 / 32
- 7. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 7 / 32
- 8. How to develop models ? DEVS Initiated in 1976 by B. Zeigler to model and simulate dynamic systems A discrete event formalism: the events drive the simulation Modular and hierarchical approach: two entities atomic and coupled models Formal approach for model coupling: The closed under coupling property: a coupled model is equivalent to an atomic Abstract simulators (algorithms) to develop simulation tools: One simulator for driving one atomic model One coordinator for driving one coupled model Extensible: DS-DEVS: dynamic-structure DEVS, to add or remove, models or connections at run-time Cell-DEVS, Fuzzy-DEVS, RT-DEVS etc. Quesnel et al. (INRA) Package system for VLE 8 / 32
- 9. Atomic model definition (DEVS) M = X , Y , S, δext , δint , δcon , ta, λ Where: δext , δint , δcon , ta, λ are functions, S the set of states Coupled model definition (DEVS) N = X , Y , D, EIC , EOC , IC Executive model definition (DS-DEVS) Mχ = Xχ , Yχ , Sχ , γ, Σ∗ , δintχ , δextχ , δconχ , λχ , τχ Where: γ and Σ∗ are structure change functions Quesnel et al. (INRA) Package system for VLE 9 / 32
- 10. How to develop models ? VLE VLE provides C++ shared library which embeds the DEVS simulation kernel, thus: users have to develop C++ class to develop atomic (DEVS) or executive models (DS-DEVS), or sub-formalisms class Dynamics { public : virtual Time timeAdvance () const ; virtual void i n t e r n a l T r a n s i t i o n ( const Time & time ); virtual void e x t e r n a l T r a n s i t i o n ( const Time & time , const E x t e r n a l E v e n t L i s t & lst ); virtual void output ( const Time & time , E x t e r n a l E v e n t L i s t & out ) const ; virtual void c o n f l u e n t T r a n s i t i o n ( const Time & time , const E x t e r n a l E v e n t L i s t & lst ); virtual Value * observation ( const O b s e r v a t i o n Ev e n t & event ) const ; } Quesnel et al. (INRA) Package system for VLE 10 / 32
- 11. How to develop models ? Sub-formalisms DEVS allows to develop sub-formalism of DEVS. With VLE, we provide: dynamics which encapsulate mathematical formalisms such as differential equations, difference equations, Petri nets, etc. structurals which manipulates the structure of the model: cellular automaton, graph of communications, etc. Dynamic sub-formalisms are atomic models with a specified behavior: we provide transition functions Structural sub-formalisms are atomic models and executive models of DS-DEVS Quesnel et al. (INRA) Package system for VLE 11 / 32
- 12. How to develop models ? For sub-formalisms We provide simplified API (object oriented development) We provide C++ generators For example, with the difference equation sub-formalism (which inherits the previous Dynamics class), only the compute function is available: class MyModel : public vle :: extension :: D i f f e r e n c e E q u a t i o n { [...] virtual double compute ( const Time & time ) { x = y ( -1) + z ( -1); y = z ( -1); } }; Quesnel et al. (INRA) Package system for VLE 12 / 32
- 13. How to develop models ? GVLE: GUI of VLE, an IDE To develop source code, the structure of the model, to assign initial conditions, observations, to define experimental frames, etc.: Quesnel et al. (INRA) Package system for VLE 13 / 32
- 14. The problem Context Modelers develop models in C++ and can reuse models from other modelers Models are stored into shared library. VLE loads and extracts atomic model from shared library Problems We clearly identify a problem of sharing and reusing code: VLE C++ API / ABI version Sub-formalism C++ API / ABI version Model change version (for example: the input or output DEVS ports) Solution To improve collaborative work under VLE we develop a package system Quesnel et al. (INRA) Package system for VLE 14 / 32
- 15. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 15 / 32
- 16. Package system We identify the following parts to develop Define the format of VLE’s package Develop and upgrade the tools and libraries of VLE (VLE, GVLE, PyVLE and RVLE) to take into account packages: Open, install, read, compile, check, remove packages Develop internet access from VLE to download and install packages from distant distributions of packages Add a checking consistency onto distributions and user’s installation This work is inspired from the FOSS (Free Open Source Software) community Quesnel et al. (INRA) Package system for VLE 16 / 32
- 17. What’s a package in VLE We need to share with the package system: Data: eg. climate data files etc. External shared library: eg. scientific libraries Sub-formalism libraries: structurals and dynamics Models: classes inherit from atomic, executive or sub-formalisms classes GUI plug-ins: to improve GVLE with new features Output plug-ins: specific output simulation, database etc. Documentation Package must have a version number, an unique name and a list of run-time and build dependencies Quesnel et al. (INRA) Package system for VLE 17 / 32
- 18. What’s a package in VLE Finite State automaton The vle.extension.fsa package provides: Harel’s state-chart, Mealy, Moore and FDDevs sub-formalisms A graphical user interface to draw automaton an generate C++ For the user of the package: Shared library which stores binary code of sub-formalisms C++ Headers of the sub-formalisms GUI plug-ins The vle.extension.fsa package, only depends of VLE 1.1 Quesnel et al. (INRA) Package system for VLE 18 / 32
- 19. Example: FSA package GUI to develop automaton Quesnel et al. (INRA) Package system for VLE 19 / 32
- 20. Example: FSA package Generated C++ source code Quesnel et al. (INRA) Package system for VLE 20 / 32
- 21. What’s a package in VLE The content and the structure of a package package / data / # data ( climate data file ) exp / # experiment file ( structure of DEVS model ) src / # source code test / # unit test lib / # shared sub - formalism libraries plugins / # simulators , GUI plug - ins shared libraries Authors . txt Description . txt # the most important file License . txt A source package does not have the lib and plugins directories A binary package does not have the src and test directories Quesnel et al. (INRA) Package system for VLE 21 / 32
- 22. What’s a package in VLE The description file Source : weather - gen Version : 1.0.0 -0 Section : agronomic Maintainer : quesnel@users . sourceforge . net Homepage : http :// www . vle - project . org Build - Depends : vle ( >= 1.1) , differential - equation ( >= 1.0) Depends : weather (= 1.2.3) , output - file ( >= 1.0) Conflicts : Description : A weather generator . The build-dependencies are required to build and compile the package The dependencies are required at run-time of the simulation The conflicts package must be removed before the compilation of the package or at run-time of the simulation Quesnel et al. (INRA) Package system for VLE 22 / 32
- 23. What’s a package in VLE An example of package dependencies with an agronomic model DifferenceEquation, Depends and build-depends of FSA, Agent, PRNG maize-optim package are sub-formalisms: indirect build-depends packages for maize-optim Optim is sub-formalism: direct build-depend package for maize-optim Other packages are run-time depends Quesnel et al. (INRA) Package system for VLE 23 / 32
- 24. Distribution Definition The model developers build models or build sub-formalisms and make upstream packages The package maintainers get, check, fix upstream packages and pull it into distribution The distribution stores all packages. A consistency check is automatically done before adding or removing a package. Users of the distribution user community can download and install distribution’s packages Users can be connected to several distributions Distributions are accessible via http or ftp protocols Quesnel et al. (INRA) Package system for VLE 24 / 32
- 25. Distribution: Workflow Quesnel et al. (INRA) Package system for VLE 25 / 32
- 26. Distribution: Workflow Quesnel et al. (INRA) Package system for VLE 25 / 32
- 27. Distribution: Workflow Quesnel et al. (INRA) Package system for VLE 25 / 32
- 28. Distribution: Workflow Quesnel et al. (INRA) Package system for VLE 25 / 32
- 29. How to check package consistency ? A distribution allows users to install and uninstall packages from a distant repository A distribution is a finite set of packages rules Each package rule is a tuple of the form (p, B, D, C ): where: p is a package, B is a set a dependency build clauses for p, D is a set of dependency clauses for p, C is a set of conflict clauses for p The dependency build clauses and dependency clauses must be present to compile and use the package p The dependency build packages can be removed after the installation Quesnel et al. (INRA) Package system for VLE 26 / 32
- 30. How to check package consistency ? Each dependency clause or dependency build clause is a disjunction of packages p1 |...|pk A dependency clause requires that some packages from the set {p1 , ..., pk } to be present Conflict clause requires that package p is not present in order to install the package p Thus, a valid installation is a subset of the packages in the distribution An installation profile is valid if dependency clauses and conflict clauses are satisfied A valid installation profile for a distribution should satisfy the dependency and conflict clauses of each package rule of the distribution Quesnel et al. (INRA) Package system for VLE 27 / 32
- 31. How to check package consistency ? The package system has two critical problems: To install the new package p given a distribution R and an installation profile P, the package system has to check if there is a profile P containing p such that P ∪ P is a valid installation profile for R It may happen that a new package p cannot be installed because it is in conflict with some packages already installed on the machine. In this case, we must first uninstall some packages before attempting to install p. The task of the package system is then to identify a set of packages P such that p can be installed on P − P To address these two problems, we can use a satisfiability problem solver (SAT) or a constraint satisfaction problem (CSP) Quesnel et al. (INRA) Package system for VLE 28 / 32
- 32. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 29 / 32
- 33. Results The major parts of the package system has been developed The major packages of VLE are converted into packages (near 20 packages) RECORD project is converting its packages in new VLE packages (near 100 packages) Package system is working to download and install packages from a server All tools of VLE are converted to package systems: VLE, GVLE, RVLE and PyVLE However, The package system does not embed a complete solver yet The distribution package does not provide consistency checking The package system of VLE is available. VLE and RECORD users use it ! Quesnel et al. (INRA) Package system for VLE 30 / 32
- 34. 1 Context 2 Problem 3 Package system 4 Results 5 Conclusions and perspectives Quesnel et al. (INRA) Package system for VLE 31 / 32
- 35. Conclusions and perspectives A success: Users are happy to use the package system to share and reuse model, sub-formalism and data The use of package seems to simplify the construction of complex models by aggregating atomic models and data from other packages Users intensively use generic models in Weather or Glue packages: these packages are now robust Package system improves the global quality of the models and simplify the maintenance of models However, The package system needs to have computer engineers: To integrate and maintain packages of the community To perform release of the distribution Quesnel et al. (INRA) Package system for VLE 32 / 32