Adaptive User Feedback for IR-based Traceability Recovery
1 like183 views
The document discusses adaptive user feedback methods for improving traceability recovery in information retrieval systems, using software artifacts as queries. It examines the effectiveness of various relevance feedback techniques and presents empirical evaluations based on three software projects, aiming to assess whether adaptive relevance feedback enhances performance over traditional methods. The results indicate that adaptive relevance feedback can significantly improve retrieval precision compared to standard approaches.
Adaptive User Feedback for IR-based Traceability Recovery
- 1. Annibale Panichella Andy ZaidmanAndrea De Lucia Adaptive User Feedback for IR-based Traceability Recovery
- 3. Search Use Case 1 Software Artefacts used as query Traceability Recovery
- 4. Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… 90% 85% 81% 74% Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … 62% Search List of candidate links Traceability Recovery
- 5. Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … ✓ ✗ ✗ ✓ ✓ Search Traceability Recovery
- 6. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006
- 7. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… 81% 74% Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … 62% Search ✓ ✗
- 8. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … ✓ ✗ Search 81% 74% 62% Learning Process
- 9. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… ✓ ✗ Search 81% 74% 62% 43% Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… 76% 77%
- 10. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… 77% 76% Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … 43% ✓ ✗ Search
- 11. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 Use Case 1 Class1.java nl.tudelft.package1.class public class Class1{ private int attribute 1; private String attribute 2; … Class5.java nl.tudelft.package2.class public class Class5{ public Class5 (int parameter1, int parameter2, String parameter3)… Class2.java nl.tudelft.package1.class public class Class2 extends Class1 { private int attribute 1; … Class63.java nl.tudelft.package5.class public class Class63{ public Class63 (Class12 parameter1, int parameter2)… Class27.java nl.tudelft.package3.class private int attribute 1; private String attribute 2; … ✓ ✗ ✓ ✓ ✗ Search
- 12. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 “Analyst feedback improves the final trace results” for requirements tracing (Standard Rocchio)
- 13. Relevance Feedback Hayes et al., “Advancing Candidate Link Generation for Requirements Tracing: the Study of Methods” IEEE Transaction on Software Engineering 2006 De Lucia et al., “Incremental Approach and User Feedback: a Silver Bullet for Traceability Recovery” International Conference on Software Maintenance, 2006 Relevance feedback does not improve and sometimes worsens the accuracy of an IR method when applied to different software artefacts (Standard Rocchio)
- 14. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008.
- 15. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 1. Queries contain few words if compared to the size of documents to retrieve Short query Web pages with hundreds of words
- 16. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 1. Queries contain few words if compared to the size of documents to retrieve Use Case Test Case Query? Query?
- 17. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 1. Queries contain few words if compared to the size of documents to retrieve In traceability, source artefacts (queries) can be more verbose than target artefacts (documents)
- 18. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 2. Cluster hypothesis: relevant documents must be similar to each others, i.e., they should cluster in the vector space Relevant Documents Non-Relevant Documents Query Term1 Term2
- 19. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 2. Cluster hypothesis: relevant documents must be similar to each others, i.e., they should cluster in the vector space Relevant Documents Non-Relevant Documents Query Term1 Term2
- 20. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 2. Cluster hypothesis: relevant documents must be similar to each others, i.e., they should cluster in the vector space Language used in software artefacts is much more homogeneous than natural language Term1 Term2
- 21. Theory behind Relevance Feedback Mannin et al, “Introduction to Information Retrieval”. Cambridge University Press, 2008. Assumption 2. Cluster hypothesis: relevant documents must be similar to each others, i.e., they should cluster in the vector space Language used in software artefacts is much more homogeneous than natural language Assumption 1. Queries contain few words if compared to the size of documents to retrieve In traceability, source artefacts (queries) can be more verbose than target artefacts (documents)
- 22. Adaptive Relevance Feedback List = initial ranked list of candidate links while not (stopping criterion) { Get the link (source, target) on top of List The user classifies (source, target) Apply the standard Rocchio to source } (Standard Rocchio)
- 23. Adaptive Relevance Feedback List = initial ranked list of candidate links while not (stopping criterion) { Get the link (source, target) on top of List The user classifies (source, target) if (source < target) Apply the standard Rocchio to source if (target < source) Apply the standard Rocchio to target } Adaptive Standard Rocchio Apply the relevance feedback only to the shortest artefacts (Assumption 1)
- 24. Adaptive Relevance Feedback List = initial ranked list of candidate links while not (stopping criterion) { Get the link (source, target) on top of List The user classifies (source, target) if (source < target && TruePositive(source) > FalsePositive(source)) Apply the standard Rocchio to source if (source < target && TruePositive(source) > FalsePositive(source)) Apply the standard Rocchio to target } Apply the relevance feedback only to the shortest artefacts (Assumption 1) Adaptive Standard Rocchio Apply the relevance feedback if and only if the number of correct links is >= to the number of false positives (Assumption 2)
- 26. Empirical Evaluation Context: three software projects
- 27. Empirical Evaluation We investigates the following research questions: RQ1: Does the adaptive relevance feedback improve the performances of the Vector Space Model? RQ2: Does the adaptive relevance feedback outperform the standard relevance feedback? Context: three software projects
- 28. Empirical Evaluation Context: three software projects We investigates the following research questions: RQ1: Does the adaptive relevance feedback improve the performances of the Vector Space Model? RQ2: Does the adaptive relevance feedback outperform the standard relevance feedback? Metrics: Precision = TP/ (TP+FP) Recall = TP / (Tot Links) Wilcoxon Test (non-parametric)
- 29. Empirical Results Easy-Clinic: tracing UC onto CC Easy-Clinic: tracing ID onto CC Easy-Clinic: tracing TC onto CC
- 30. Empirical Results i-Trust: tracing UC onto JSP Modis: tracing HLR onto LLR
- 31. Empirical ResultsAveragePrecision 0 23 45 68 90 UC-CC ID-CC TC-CC i-Trust Modis VSM RF Adaptive RF Statistical Significance: AdaptiveRF > VSM = 4/5 AdaptiveRF > RF = 4/5 RF > VSM = 1/5 Wilcoxon test