Experimenting with eXtreme Design (EKAW2010)
- 1. Experimenting with eXtreme Design Eva Blomqvist, Valentina Presutti, Enrico Daga, and Aldo Gangemi STLab, ISTC-CNR, Rome, Italy EKAW 2010 Lisbon (Portugal) – 2010-10-12
- 2. Outline Background on Content ODPs Problem – Research Questions XD Tools and XD methodology Experiment setup Results of analysis Confirming previous results XD Tools XD Methodology Conclusions and future work
- 3. Background - ODPs It is difficult to construct good quality ontologies! Lack of guidelines Best-practices exist but how can we communicate them? Classes of problems in ontology design that can be solved by applying common solutions => Ontology Design Patterns (ODPs) Content ODPs (CPs) small ontologies with explicit documentation of design rationales modeling “good practices” An ontology can be seen as a… Composition of CPs + Dependencies + Expansion
- 4. Background – ODPs (cont.) A Content ODP (CP) is always associated with requirements, usually expressed using Competency Questions (CQs) Example: InformationRealization What are the physical realizations of this information object? What information objects are realized by this physical object?
- 5. Background – ODPs (cont.) Collected at http://www.ontologydesignpatterns.org Currently: Logical-, Architectural-, Content-, Re-engineering-, Alignment-, and Lexico-syntactic ODPs
- 6. Problem Overall question: What are the benefits of ODPs in ontology engineering? (Initial focus on Content ODPs) Study conducted in 2008-09 (reported at K-CAP’09) Are Content ODPs perceived as useful? Yes! Are the ontologies constructed using Content ODPs ‘better’, in some modelling quality sense? Coverage of problem decreased (slower?) but major improvement in usability aspects, and fewer common mistakes Are the tasks given to the participants solved faster when using Content ODPs? Not really, rather slower (too little experience?) How do participants use the Content ODPs provided, and what support would be beneficial? How to find and select ODPs? How to reuse them? Tools?
- 7. Problem (cont.) Since the previous study, we have developed: The eXtreme Design (XD) methodology XD Tools Summary of research questions: Can we confirm the results from the previous study (Questions 1-4)? + How is modularity affected? Does XD Tools support the process of reusing CPs? Does the XD methodology support the process of reusing CPs, and does it affect any of the aspects from the previous study (e.g., time, quality)?
- 8. Related Work Ontology Engineering methodologies Classical methodologies are similar to software engineering methodologies Few focus on collaboration, and very few are pattern-based – (Clark & Porter, 1997) and (Maas & Janzen, 2009) Most are not evaluated, just motivated through… Example use case Theoretical comparison Pattern-based tools Mainly for Logical ODPs (e.g. support from OPPL, and templates and wizards in Protégé) Tool evaluation mostly as benchmarking/comparison
- 9. XD Tools Plugin for Eclipse-based OE environments (TopBraid Composer, NeOn Toolkit, etc.) Main parts Registry browser XD Selector Specialization wizard Annotation dialogue XD Analyzer
- 10. XD Tools Browse, search, and get Content ODPs Analyze your ontology against good practices and patterns Specialize, compose, annotate ODPs and ontologies
- 11. XD Methodology An agile approach to ontology engineering Two sets The problem space, i.e., the actual modeling issues (local problems) The solution space, i.e., reusable modeling solutions Test-driven: unit tests on modules – integration tests on overall ontology Task-focused: solve only and exactly the requirements at hand Divide-and-conquer (design pairs) – collaboration and integration are crucial aspects
- 12. XD Methodology (cont.) … Step 5 - Select a coherent set of CQs, treating one modelling issue, for development iteration. Step 6 - Match the CQs to CPs, candidate CPs for reuse are identified. Step 7 - Select CPs to use: best fit to local problem without unnecessary overhead. Step 8 - Reuse (import, specialize) and integrate (compose, extend) selected CPs. Step 9 - Test (unit tests, e.g., through SPARQL queries) and fix. Then start the next iteration. …
- 13. Experiment Setup 2 sessions, 35 participants in total (inexperienced, mainly master and PhD students, and junior researchers) Controlled settings: same tools, same training and instructions, same tasks, same patterns Procedure: One difference: Session 2 – XD Tools more mature
- 14. Experiment Setup (cont.) Task 1: Context The national association for promotion of theater in Italy wants to set up a web-based system for keeping track of details about theater productions and the actors at different theaters. In order to support reasoning about the productions, the system should be based on an ontology. … Story: theater productions During each year a number of theatre festivals are held in cities around Italy. In January 2007 a festival called “Roma Loves Shakespeare” took place in Rome. Two different productions of “The Merchant of Venice” participated, one from a theatre in Pisa and the other from a theatre institute in Venice, featuring an ensemble of university art students. Other plays were Othello and a Midsummer Night’s Dream … Competency questions (CQs) and contextual statements of theater production When did a certain theatre festival take place? Where did a certain festival take place? What plays could be seen during a certain theatre festival? In what city is a certain theatre located? In what country is a certain city located? What play is the basis of a certain production? … Contextual statements: A production has exactly one premier.
- 15. Evaluation of Results Analysis of questionnaire responses Evaluation of the constructed ontologies (Mainly functional and usability evaluations – see K-CAP’09 paper) Coverage of problem: terminological coverage and task coverage Usability: e.g., presence of labels, comments, inverse relations, disjointness, level of axiomatization Modelling mistakes: incomplete solutions, solutions with shortcomings Pattern usage: implicit and explicit
- 16. Results – Confirming previous conclusions? Are CPs perceived as useful by the participants? Confirmed – Increase for second session: Due to XD Tools? Are the ontologies constructed using CPs ‘better’, in some modelling quality sense? Coverage: Reduction of terminological coverage is no longer detected – Due to XD Tools? Usability: Confirmed – Most prominent improvement! Are the tasks solved faster when using CPs? With tool support: no longer slower! What common modelling ‘mistakes’ can be identified, when not using patterns and when using CPs? Decrease in occurrence of most frequent mistakes confirmed (44% average decrease) - Same types of mistakes
- 17. Results - Modularity Do CPs increase the modularity of ontologies? Task 1: no ontologies are modularized Task 2: the ontologies contain on average 7.5 modules Conclusion: Since the participants choose to reuse the CPs as OWL-modules , rather than ideas for solutions, this inherently introduces modularity
- 18. Results – XD Tools How well does XD Tools support… …finding CPs to reuse? …specializing CPs? Does XD Tools introduce overhead?
- 19. Results – XD Tools How well does XD Tools support… …finding CPs to reuse? …specializing CPs?
- 20. Results – XD Tools How well does XD Tools support… …finding CPs to reuse? …specializing CPs? Does XD Tools introduce overhead?
- 21. Results – XD methodology Is the XD methodology perceived as useful by the participants? Is the XD methodology a ‘natural’ way to work with CPs? Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs? Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs? What common modelling ‘mistakes’ can be identified and are they different from the ones noted when ‘only’ using CPs?
- 22. Results – XD methodology Is the XD methodology perceived as useful by the participants? 94% claimed to follow the XD methodology closely
- 23. Results – XD methodology Is the XD methodology perceived as useful by the participants? Is the XD methodology a ‘natural’ way to work with CPs?
- 24. Results – XD methodology Is the XD methodology perceived as useful by the participants? Is the XD methodology a ‘natural’ way to work with CPs? Are the ontologies constructed using the XD methodology ‘better’, in some modelling quality sense, than the ontologies constructed ‘only’ using CPs? Small increase from Task 2 to Task 3 (could be due to maturation) Are the tasks solved faster when using the XD methodology, compared to ‘only’ using CPs? Small increase – however, mistakes seem to be found more easily, hence some parts seem to be done faster!
- 25. Results – XD methodology … What common modelling ‘mistakes’ can be identified and are they different from the ones noted when ‘only’ using CPs? Continued decrease in occurrence of most frequent mistakes from Task 2 to 3 (15% average decrease) Two types of errors decrease significantly more than the others: N-ary relations – decrease by 64% Missing datatype properties – decrease by 46% Conclusion: XD is test-driven – structured unit testing leads to finding certain types of mistakes
- 26. Future Work Currently ongoing Development of the integration and collaboration part of the XD methodology Explorative experiments – 1 session performed From experimenting on the iteration of the design pairs to experimenting on the complete development team Things to be considered Collaboration between pairs – integration pair? Integration testing and refactoring Information feedback loop … Planned experiment Comparative study of XD and other methodologies
- 27. Future Work (cont.) Additional experimental work Other domains and tasks New target groups – experts Other types of patterns Presentation and representation issues Analysing future ontologies (when patterns are more wide-spread) XD Tools Shortcuts for expert users to reduce overhead Improve search functionalities XD methodology Elaborate collaboration and integration aspects
- 28. Conclusions Confirmed conclusions Content ODPs are perceived as useful They improve mainly the usability aspects of the ontologies, but also reduces the occurrence of a number of common ‘mistakes’ Development is neither faster nor slower XD Tools Supports the use of CPs – no longer slower Perceived as useful, although adds some overhead XD methodology Perceived as useful, and it is a natural way of working with CPs (descriptive and pragmatic) Main advantage seems to be testing – drastically decreases some common ‘mistakes’