MoDELS'16 presentation: Integration of a Graph-Based Model Indexer in Commercial Modelling Tools
2 likes891 views
Modelio M2T with Jython HT: Hawk M2T with EGL A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 23 / 27 Intro Integration Evaluation Conclusions M2T with Hawk: conclusions Indexing time grows linearly with model size Generation time grows logarithmically Hawk+EGL is faster than Modelio+Jython for large models Hawk scales better due to indexing and querying optimizations Modelio+Jython has higher constant overhead Break-even point is ~10k elements For large projects, Hawk is preferable SOFTEAM now uses Hawk for all M2
MoDELS'16 presentation: Integration of a Graph-Based Model Indexer in Commercial Modelling Tools
- 1. Intro Integration Evaluation Conclusions Integration of a Graph-Based Model Indexer in Commercial Modelling Tools Antonio García-Domínguez, Konstantinos Barmpis, Dimitrios S. Kolovos, Marcos Aurelio Almeida da Silva, Antonin Abherve and Alessandra Bagnato Aston University, University of York, SOFTEAM Cadextan MoDELS’16 October 7th, 2016 A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 1 / 27
- 2. Intro Integration Evaluation Conclusions Outline 1 Intro 2 Integration 3 Evaluation 4 Conclusions A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 2 / 27
- 3. Intro Integration Evaluation Conclusions Motivation for this work MONDO EU project: produced Hawk indexer Goal: tackle scalability of MDE in its various facets One product was Hawk, a scalable model indexer: https://github.com/mondo-project/mondo-hawk Hawk mirrors fragmented models into NoSQL graphs SOFTEAM: Constellation collab server needed search SOFTEAM has 20+ years experience on UML tooling Modelio is their open-source modelling tool Joined MONDO due to scalability challenges Created Constellation: Modelio collab server Constellation had no search box – need indexing! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 3 / 27
- 4. Intro Integration Evaluation Conclusions Motivation for this work MONDO EU project: produced Hawk indexer Goal: tackle scalability of MDE in its various facets One product was Hawk, a scalable model indexer: https://github.com/mondo-project/mondo-hawk Hawk mirrors fragmented models into NoSQL graphs SOFTEAM: Constellation collab server needed search SOFTEAM has 20+ years experience on UML tooling Modelio is their open-source modelling tool Joined MONDO due to scalability challenges Created Constellation: Modelio collab server Constellation had no search box – need indexing! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 3 / 27
- 5. Modelio: SOFTEAM’s modelling tool Open source, scriptable, UML/BPMN/SysML... Structure: workspace → projects → fragments (.exml) Projects can import .ramc libraries (zipped fragments)
- 6. Intro Integration Evaluation Conclusions Constellation: SOFTEAM’s collab server Admin server coordinates agents and provides web UI Agent nodes (HTTP/SVN) access/host fragments No querying facilities A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 5 / 27
- 7. Intro Integration Evaluation Conclusions Hawk: NoSQL model indexer We go from these model files... A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27
- 8. Intro Integration Evaluation Conclusions Hawk: NoSQL model indexer ... to these NoSQL graphs. A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27
- 9. Intro Integration Evaluation Conclusions Hawk: NoSQL model indexer EPackages → metamodel nodes EClasses → type nodes A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27
- 10. Intro Integration Evaluation Conclusions Hawk: NoSQL model indexer .xmi → file nodes EObjects → element nodes A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27
- 11. Intro Integration Evaluation Conclusions Hawk: NoSQL model indexer MM index: EPackage URI → metamodel node File index: repo + file path → file node Users can define custom indices by attribute/expression A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 6 / 27
- 12. Intro Integration Evaluation Conclusions Outline 1 Intro 2 Integration 3 Evaluation 4 Conclusions A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 7 / 27
- 13. Intro Integration Evaluation Conclusions Integration approach Key detail in Hawk: plugin-based architecture Hawk has abstraction layers for: Database backends (Neo4j, OrientDB) Model persistence formats (XMI, UML, Modelio...) Query languages (Epsilon Object Language) Steps taken 1 Expose Modelio metamodels to Hawk 2 Write .exml/.ramc parser 3 Extend Hawk with Modelio specifics 4 Embed into Constellation A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 8 / 27
- 14. Intro Integration Evaluation Conclusions Initial legal issues Problem Hawk is EPL, Modelio is GPLv3: incompatible licenses Cannot reuse neither metamodel code nor parsing code Solutions SOFTEAM wrote M2T transformation to expose metamodels as Apache-licensed library (ModelioMetamodelLib) http://github.com/aabherve/modelio-metamodel-lib Hawk includes transformation to Ecore Hawk devs collaborated with SOFTEAM to write new parsers Clean-room approach: no looking at Modelio code! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 9 / 27
- 15. Intro Integration Evaluation Conclusions Initial legal issues Problem Hawk is EPL, Modelio is GPLv3: incompatible licenses Cannot reuse neither metamodel code nor parsing code Solutions SOFTEAM wrote M2T transformation to expose metamodels as Apache-licensed library (ModelioMetamodelLib) http://github.com/aabherve/modelio-metamodel-lib Hawk includes transformation to Ecore Hawk devs collaborated with SOFTEAM to write new parsers Clean-room approach: no looking at Modelio code! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 9 / 27
- 16. .exml format <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT>
- 17. .exml format: version, dependencies <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Format version Dependencies
- 18. .exml format: version, dependencies <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Format version Dependencies
- 19. .exml format: root object <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Object name, metaclass, UID Same UID = same object Happens when importing libraries Container UID EMF compatibility would require deriving containments
- 20. .exml format: root object <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Object name, metaclass, UID Same UID = same object Happens when importing libraries Container UID EMF compatibility would require deriving containments
- 21. .exml format: attributes and dependencies <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Attributes: name-value pairs Dependencies: references COMPositions (containment?) LINKs (non-containment)
- 22. .exml format: attributes and dependencies <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> Attributes: name-value pairs Dependencies: references COMPositions (containment?) LINKs (non-containment)
- 23. .exml format: nested objects <EXT object="Component" version="3"> <DEPS> <ID name="Component" mc="Class" uid="2d7b..."/> <EXTID name="Element" mc="Class" uid="4ed7..."/> </DEPS> <OBJECT> <ID name="Component" mc="Class" uid="2d7b..."/> <PID name="Architecture" mc="Package" uid="ea87..."/> <ATTRIBUTES> <ATT name="Name">Component</ATT>... </ATTRIBUTES> <DEPENDENCIES> <COMP relation="OwnedOperation">...</COMP> <COMP relation="Parent">...</COMP> </DEPENDENCIES> </OBJECT> </EXT> <OBJECT> <ID name="..." mc="Generalization" uid="58e6..."/> <PID name="Component" mc="Class" uid="2d7b..."/> <ATTRIBUTES>...</ATTRIBUTES> <DEPENDENCIES> <LINK relation="SuperType"> <ID name="Element" mc="Class" uid="4ed7..."/> </LINK> </DEPENDENCIES> </OBJECT> PID here is XML container’s UID: not always the case! LINK to object in another .exml
- 24. Intro Integration Evaluation Conclusions Hawk: tweaks needed for Modelio References are UID-based Parsed objects/refs can declare they are UID-based If so, Hawk will maintain/use UID → object index Same UID = same object Parsed resources can declare they provide “singletons” If so, Hawk will merge objects with same UID into 1 node A single element node can belong then to multiple files Container-first rather than containment-first Optional automated derivation of EMF-style containments Only needed for compatibility with EMF-based tools Not needed for Epsilon compatibility A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 15 / 27
- 25. Intro Integration Evaluation Conclusions Outline 1 Intro 2 Integration 3 Evaluation 4 Conclusions A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 16 / 27
- 26. Intro Integration Evaluation Conclusions Indexing costs on real projects (Neo4j) File type .exml .ramc Minimum 229B 23kB Median 1.51kB 8.45MB Maximum 2.77MB 30.98MB Count 201 526 23 Stages 2GB 4GB 8GB Frag ins. 2554s 2502s 2371s Frag conn. 728s 541s 443s Cont deriv. 1851s 1234s 1123s Total 5171s 4287s 3958s Inputs SOFTEAM provided 15 in-house projects 2GB between .exml and .ramc, 1.23M elems Memory consumption W/containment: 2GB+ Without: 1GB (3146s) Diminishing returns A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 17 / 27
- 27. Embedding into Constellation: advanced queries Required new OrientDB backend (Neo4j is GPLv3-licensed) SOFTEAM embedded Hawk into web UI in 3 months First result: EOL query console
- 28. Embedding into Constellation: dashboards SOFTEAM used Hawk queries to feed project dashboards Regular users do not need to know any queries
- 29. Intro Integration Evaluation Conclusions M2T with Hawk: motivation Business case M2T in Modelio was done through Jython scripting SOFTEAM wanted M2T agents in Constellation Code generation as a service Which is more efficient? Modelio + Jython? Hawk + Epsilon Generation Language? Need to take into account indexing times as well! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 20 / 27
- 30. Intro Integration Evaluation Conclusions M2T with Hawk: inputs Script size param m = 1 m = 2 m = 4 m = 8 m = 16 Project size 80MB 150MB 279MB 814MB 1.8GB .exml size 58MB 104MB 179MB 422MB 752MB .exml count 1 255 2 087 3 430 7 849 13 790 Element count 47 981 88 451 154 281 367 840 657 228 SOFTEAM wrote simple UML model generator in Jython Size parameter controls number of UML classes and number of attributes/references per class A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 21 / 27
- 31. Intro Integration Evaluation Conclusions M2T with Hawk: indexing times 1 2 3 4 5 6 ·105 200 400 600 800 1,000 1,200 Project size (model elements) Indexingtime(s) 500 1,000 1,500 2,000 2,500 OrientDBdiskspaceusage(MB) Time Disk space A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 22 / 27
- 32. Intro Integration Evaluation Conclusions M2T with Hawk: generation times 1 2 3 4 5 6 ·105 100 101 102 103 Project size (model elements) Generationtime(s) MT (run 1) MT (runs 2–5) HT (run 1) HT (runs 2–5) MT: Modelio M2T HT: Hawk M2T Cold runs: 10-100x faster on Hawk Warm runs: Hawk scales better A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 23 / 27
- 33. Intro Integration Evaluation Conclusions M2T with Hawk: tradeoff 1 2 3 4 5 6 ·105 0 5 10 15 20 1 Project size (model elements) MT/(HTrun1+indexing) How many cold Modelio runs before Hawk makes more sense? For the largest project, the answer is 1! A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 24 / 27
- 34. Intro Integration Evaluation Conclusions Outline 1 Intro 2 Integration 3 Evaluation 4 Conclusions A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 25 / 27
- 35. Intro Integration Evaluation Conclusions Conclusions and future work Obtained results SOFTEAM wanted querying for their model collab server Hawk is now powering it: Licensing issues guided many design decisions Available as dashboards and advanced manual querying Hawk+EGL faster than Modelio+Jython for M2T-as-service Future work SOFTEAM: integrate Hawk in more places and provide querying capabilities for less technical users Further performance improvements in Hawk: Horizontal scalability (more RAM had diminishing returns) On-demand derived features to reduce indexing time Optimize OrientDB backend (much work done for v1.1!) A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 26 / 27
- 36. End of the presentation Questions? @antoniogado A García-Domínguez et al. Integration of a Graph-Based Model Indexer in Commercial Modelling Tools 27 / 27