Traceability - Structural Conformance Checking with Design Tests: An Evaluation of Usability and Scalability
1 like314 views
The document presents a study on design tests and the DesignWizard tool aimed at ensuring code quality by checking adherence to design rules. It evaluates the usability of the DesignWizard API through an experiment involving developers and highlights weaknesses identified in the API. The findings suggest potential improvements for the DesignWizard API and emphasize the importance of maintaining structural constraints in software development.
Traceability - Structural Conformance Checking with Design Tests: An Evaluation of Usability and Scalability
- 1. Motivation Design Test DesignWizard Evaluation Conclusion Design Tests João Brunet Dalton Guerrero and Jorge Figueiredo Federal University of Campina Grande (UFCG), Brazil. 09/28/2011 João Brunet 1/ 26
- 2. Motivation Design Test DesignWizard Evaluation Conclusion Agenda 1 Motivation 2 Design Test 3 DesignWizard 4 Evaluation API Usability 5 Conclusion João Brunet 2/ 26
- 3. Motivation Design Test DesignWizard Evaluation Conclusion Motivation Check code against design rules is an important activity to guarantee quality on source code Design Review Manual Process Error-prone Does not scale João Brunet 3/ 26
- 4. Motivation Design Test DesignWizard Evaluation Conclusion Motivation Check code against design rules is an important activity to guarantee quality on source code Design Review Manual Process Error-prone Does not scale João Brunet 3/ 26
- 5. Motivation Design Test DesignWizard Evaluation Conclusion The Problem Gap between state-of-the-art and the state-of-the-practice Approaches from the state-of-the-art require learning different languages to express design rules João Brunet 4/ 26
- 6. Motivation Design Test DesignWizard Evaluation Conclusion The Problem Lack of structural conformance checking Design decisions are violated João Brunet 5/ 26
- 7. Motivation Design Test DesignWizard Evaluation Conclusion Design Test Design Test A test that checks whether an implementation complies with a given design rule expressed as an algorithm. João Brunet 6/ 26
- 8. Motivation Design Test DesignWizard Evaluation Conclusion What’s the big deal? It is written in the target programming language Automated test João Brunet 7/ 26
- 9. Motivation Design Test DesignWizard Evaluation Conclusion Example of Design Rule Only controller package shall access dao package of a given component. João Brunet 8/ 26
- 10. Motivation Design Test DesignWizard Evaluation Conclusion Example of Design Test João Brunet 9/ 26
- 11. Motivation Design Test DesignWizard Evaluation Conclusion Example of Design Test Facts about the code (an API) João Brunet 10/ 26
- 12. Motivation Design Test DesignWizard Evaluation Conclusion Example of Design Test Facts about the code (an API) Assertion routines João Brunet 11/ 26
- 13. Motivation Design Test DesignWizard Evaluation Conclusion DesignWizard João Brunet 12/ 26
- 14. Motivation Design Test DesignWizard Evaluation Conclusion Some methods of DesignWizard’s API getAllPackages() getAllSubClasses() getStaticMethods() getImplementedInterfaces() ... João Brunet 13/ 26
- 15. Motivation Design Test DesignWizard Evaluation Conclusion Process João Brunet 14/ 26
- 16. Motivation Design Test DesignWizard Evaluation Conclusion Process João Brunet 15/ 26
- 17. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Methodology “It is the comparison between what developers expect and what the API provides that is interesting when evaluating the usability of an API (Steven Clarke)” Think Aloud Protocol For APIs to capture expectations, goals, difficulties and strategies João Brunet 16/ 26
- 18. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Experiment 11 developers 5 real design tests (selected by the designer of the OurGrid) OurGrid Eclipse API Pseudocode vs. Code João Brunet 17/ 26
- 19. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Experiment João Brunet 18/ 26
- 20. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Experiment Only one developer did not write one instance of design test. João Brunet 19/ 26
- 21. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Experiment ...but, 10 tests were incorrectly written. The good news is that all developers found the right method, but used it incorrectly. João Brunet 20/ 26
- 22. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Results Two weaknesses of API: getAllClasses() We decided to follow the Java Reflection API pattern getDeclaredMethod (String name) An appropriated name of the parameter would be signature João Brunet 21/ 26
- 23. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Results Think Aloud Protocol revealed that devolopers had no difficulties to find the right method to achieve a given task Results suggested changes to DesignWizard API and documentation. João Brunet 22/ 26
- 24. Motivation Design Test DesignWizard API Usability Evaluation Conclusion Threats to validity Coding and designing expertise of developers More complex design rules may produce different observations The API was not fully evaluated João Brunet 23/ 26
- 25. Motivation Design Test DesignWizard Evaluation Conclusion Final Remarks Approach to check conformance between code and design rules Evaluation regarding its usability Design Tests and maintenance Executable documentation of design rules Enables developers changing functionality keeping in mind structural constraints João Brunet 24/ 26
- 26. Motivation Design Test DesignWizard Evaluation Conclusion Thank you www.designwizard.org João Brunet 25/ 26
- 27. Motivation Design Test DesignWizard Evaluation Conclusion Related Work ArchJava Reflexion Models FindBugs ... João Brunet 26/ 26