Open-Source Formative Evaluation Process in Remote Software Maintenance
2 likes698 views
The document outlines the FastFix project, an open-source initiative aimed at developing a monitoring control platform for remote software maintenance through an agile development and formative evaluation process. It details phases of evaluation, tools used, performance requirements, and metrics for success, such as code quality and issue management. The project seeks to improve software development by identifying and addressing issues throughout the project lifecycle.
![Metrics until 15th May 2012
Code analysis
• Code quality
• 34.4% Sonar Rules Compliance Index
• 2547 violations [ 1 blocker, 147 critical, 1610 major, 732
minor, 57 info]
• 23.4% comments, 8.4% duplications, 100% test success
46/49](https://image.slidesharecdn.com/rcis2012-fastfix-osformativeevaluation-120518052121-phpapp02/85/Open-Source-Formative-Evaluation-Process-in-Remote-Software-Maintenance-46-320.jpg)
Open-Source Formative Evaluation Process in Remote Software Maintenance
- 1. FastFix Open-Source Formative Work Plan Evaluation Process in Remote Software Maintenance Alessandra Bagnato, TXT e-solutions – Corporate Research Javier Cano and Christophe Joubert, Prodevelop May 18th 2012 Industrial Track at RCIS 2012, Sixth International Conference on Research Challenges in Information Science 2012 “Be Fit, Fast and Alert when testing, maintaining or solving bugs of your Future Internet applications” FITTEST & FastFIX & ALERT & Io.Test Joint Workshop
- 2. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 2/49
- 3. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 3/49
- 4. 1.- PROJECT OVERVIEW European project FASTFIX (ICT- 258109, June 2010 - February 2013) develops a monitoring control platform for remote software maintenance. Within this project, we have been working on a formative evaluation process that evaluates methods, concepts and prototypes as the project is being developed. 4/49
- 5. Project Overview WP 1 Project Management and Quality Assurance WP 2 FastFix conceptual framework WP 3 Context observation and User modeling WP 4 Event correlation WP 5 Fault replication WP 6 Patch generation and self-healing WP 7 Performance, security and privacy WP 8 Trials and validation Formative Evaluation is part of WP 9 Dissemination and exploitation WP8 5/49
- 6. Formative Evaluation PURPOSE to ensure that the goals of the under-development software are being achieved and to improve the software, if necessary, by means of identification and subsequent remediation of problematic aspects. 6/49
- 7. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 7/49
- 8. 2.- FORMATIVE EVALUATION PROCESS 1. Analysis 2. Metrics 3. Tools 4. Testing 5. Monitoring 6. Reporting 8/49
- 9. FORMATIVE EVALUATION PROCESS Analyze the Select metrics project and to collect. development process. Define who is Select tools to perform responsible for metrics collection. monitoring. Define tests to execute. Select the Define which tool will reports to create collect what. from the data. 9/49
- 10. Analyze the project Find the basic natures that define the project. In the case of FastFix : Strong focus in research Development Integration Distributed of new into common teams Software platform Artifacts 10/49
- 11. Analyze development process Analyze the development process to use. Try and find points that are worth monitoring. In the case of FastFix : Iterative process: Requirements change as development advances. Requirements and issues appear at each iteration. Testing is to start early and happen often. 11/49
- 12. Select interesting metrics Find interesting metrics to be gathered. In the case of FastFix : tickets opened, solved, closed, overdue, etc. committers activity, evolution of code base, regular activity. builds and tests stability. reports on ranked issues found in source code quality. 12/49
- 13. Find ‘invisible’ metrics There exist metrics that indicate a good development of the project but are not directly visible : Issues Detection of discussed issues before among they promote partners. to a problem. Responses and Honoring actuations on deadlines. comments from others. 13/49
- 14. Points to hook the formative evaluation Select tools that integrate into the development process. In the case of FastFix : Monitorization target Monitoring tool Source code quality. Static code analysis. Components integration to form the Continuous integration. FastFix platform. Automated builds and tests Tests execution. execution. Platform integration in target Integration trials after internal applications. releases. Open and resolved issues. Issues statistics reports. 14/49
- 15. Non-invasive tools to gather metrics Automate the process of metrics gathering as much as possible : Integration into existing tools. Tools invisible until Periodic jobs in build servers. notifications are necessary Periodic scripts that collect information. Automated tools collect metrics silently Periodic scripts that generate reports. 15/49
- 16. Start testing early, test often The best formative evaluation is the one that includes all stakeholders in the development : • Include end users as soon as possible. • Iterative testing with small increments. • Make release cycles short. Project execution Testing and Trials M0. M18. M33. Project starts. Testing starts. Internal Releases Project ends. 16/49
- 17. Monitor evolution, watch for deviations Monitor metrics gathered periodically (daily) : Look for • I.e: continuous integration deviations failing last week. Take • I.e: review build system Corrective logs to correct errors. actions • I.e: set up automated Use notifications to automated notifications responsibles of latest changes when build fails. 17/49
- 18. Create reports Collect all metrics in a single report • LOC • # of tests • Code quality Highlight areas to improve • Build stabilty • Tests Highlight good areas • Code quality • Dependencies management 18/49
- 19. Benefits The overall benefit of the process is to correct problems and to include new requirements before the production phase of the software project and during the whole life of the project. 19/49
- 20. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 20/49
- 21. 3.- AGILE DEVELOPMENT PROCESS (SCRUM) Development: sprints from backlog. Produces an increment of the platform. 24 h 30 days Product Sprint backlog backlog Product increment Sprint 21/49
- 22. Agile development process Development: sprints Development from backlog. 24 h Produces an increment 30 days of the platform. Product Sprint Product backlog backlog increment Testing of produced Sprint platform. Gathering of metrics. New Evaluation definition Integration requirements Bugs FastFix Test execution Data Integration in on FastFixed Metrics collection Change target system system aquisition requests revision Metrics aquisiton Formative evaluation Continuous evaluation Feedback Evaluation Summative Evaluation: Bug and evaluation enhancement issues stored in backlog. Metrics analyzed. 22/49
- 23. Agile development at a glance A simplified view of the development process followed in FastFix: Sprint planning Evaluation Development Product Testing increment 23/49
- 24. Specific tools The formative evaluation hooks into the development process using standard tools: Issues management Code repository analysis Trac Sprint Subversion planning Evaluation Development Code quality analysis Metrics collection Testing Product Sonar increment Jmeter Continuous integration server Hudson 24/49
- 25. AGILE DEVELOPMENT PROCESS Development: sprints from backlog. Produces an increment of the platform. Testing of produced platform. Gathering of metrics. Evaluation: Bug and enhancement issues stored in backlog. Metrics analyzed. 25/49
- 26. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 26/49
- 27. 4.- OPEN SOURCE SUPPORT TOOLS • FOCUS ON OPEN SOURCE • CRITERIA • SELECTED TOOLS 27/49
- 28. Focus on Open Source WHY OPEN SOURCE TOOLS ? As a research project with focus on Open Source: • Can deliver industrial level performance. • Free as in beer and speech. • Minimum investment. • Great customization. • Huge integration. 28/49
- 29. Criteria Source Code Building and Testing Alternatives Quality Issues Consortium selects among all alternatives according to: Project • Capabilities of each tool Management • Maturity of each tool • Integration with other tools • Familiarity among partners Communication channels 29/49
- 30. Subversion as source code and Maven as project management Source Code Mature, well known source control management. Integrates with lots of other tools. http://subversion.tigris.org/ Project Management Mature, well known project management, specially in Java. Integrates with lots of other tools. Simplifies dependency management and build process. http://maven.apache.org/ 30/49
- 31. Hudson as continuous integration Building and Testing • Off-the-shelf Integration with Subversion and Maven. • Notifications for failed builds. http://hudson-ci.org/ 31/49
- 32. Sonar as quality analyzer Quality • Quality analysis with FindBugs. • Integration with Hudson. • Reports generation. http://www.sonarsource.org/ 32/49
- 33. Trac as issue manager Issues • Integration with SVN. • Notifications. • Milestone planning. • Reports generation. http://trac.edgewall.org/ 33/49
- 34. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 34/49
- 35. 5.- PERFORMANCE REQUIREMENTS GOAL INVESTMENT PROCESS • Usability and • FastFix invests • Each iteration performance part of the release testing of FastFixed effort in collects systems not tunning the metrics to noticeably performance evaluate affected. of the performance. platform. 35/49
- 36. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 36/49
- 37. 6.- FASTFIX EVALUATION AT WORK Samples from the deployed tools: • Hudson • Sonar • Trac 37/49
- 38. FastFix Formative evaluation at work: HUDSON • HUDSON continuous integration system schedules builds every night of the whole platform. Notifications in place for interested parties. 38/49
- 39. FastFix Formative evaluation at work: SONAR • SONAR code analysis system analyses code just after every correct build. SONAR Components 39/49
- 40. FastFix Formative evaluation at work: TRAC • TRAC issues management system is used as a planning tool, to define milestones and their tasks, to manage responsibles for tasks and internal releases. 40/49
- 41. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 41/49
- 42. 7.- METRICS UNTIL 15TH MAY 2012 • Lines Of Code • Developers • Continuous integration • Issues management 42/49
- 43. Metrics until 15th May 2012 Lines Of Code Lines Of Code : 879742 Developers: 18 43/49
- 44. Metrics until 15th May 2012 Continuous integration Continuous integration: • Build stability: Common Unstable Client Server Unstable 0% Unstable 5% 15% Failed Failed 0% Failed 20% 20% Stable Stable Stable 80% 85% 75% Correct • Test stability: Failed Skipped 44/49
- 45. Metrics until 15th May 2012 Issues Management Issues management: • 282 tickets managed. • 170 closed tickets. • 112 open tickets. • 29 tickets in the backlog • 6 tickets from internal testing in industrial partners Other Closed Active 28% Backlog 60% 40% 10% Testing 2% 45/49
- 46. Metrics until 15th May 2012 Code analysis • Code quality • 34.4% Sonar Rules Compliance Index • 2547 violations [ 1 blocker, 147 critical, 1610 major, 732 minor, 57 info] • 23.4% comments, 8.4% duplications, 100% test success 46/49
- 47. OVERVIEW 1. Project Overview 2. Open Source Formative Evaluation Phases 3. Agile Development Process (Scrum) 4. Open Source Supporting Tools 5. Performance Requirements 6. FastFix Formative evaluation at work 7. Current metrics 8. Next steps 47/49
- 48. 8.- NEXT STEPS Execution of Collection of Reports that trials in all metrics summarize industrial data, both the partners with qualitative evaluation of live and the project. production quantitative. applications. 48/49
- 49. Alessandra Bagnato Corporate Research Division - TXT e-solutions S.p.A alessandra.bagnato@txtgroup.com Francisco Javier Cano and Christophe Joubert Prodevelop fjcano@prodevelop.es, cjoubert@prodevelop.es FastFix Project Web Site: https://www.fastfixproject.eu Thank you for your attention!