TAO: Turing test As Objective function
1 like1,257 views
The document discusses using artificial intelligence techniques like SBSE (Search-Based Software Engineering) to solve software engineering problems. It proposes that some objective functions used in SE are "surrogate" objectives that may be better learned than predefined, and that the goal should be passing a "Turing Test" of human judgment rather than optimizing a concrete metric. It issues a challenge to directly contribute AI/SBSE-generated code/patches to open source projects to get real-world feedback, rather than static interaction.
TAO: Turing test As Objective function
- 1. 26 May 2013, RAISE Workshop TAOism Shin Yoo/Centre for Research on Evolution Search and Testing, UCL, UK
- 2. MyWorld
- 6. MyWorld SBSE Meta-heuristic Optimisation Software Engineering SBSE = Representation + Objective Function + Optimisation
- 7. Test Data Generation Software Modularisation
- 8. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Software Modularisation
- 9. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| Software Modularisation
- 10. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| This is concrete: the problem defines the objective function directly and concretely. Software Modularisation
- 11. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| This is concrete: the problem defines the objective function directly and concretely. Software Modularisation “I want my 892 classes to be properly modularised” (insert your ugliest classes here)
- 12. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| This is concrete: the problem defines the objective function directly and concretely. Software Modularisation “I want my 892 classes to be properly modularised” (insert your ugliest classes here) Objective Function ?
- 13. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| This is concrete: the problem defines the objective function directly and concretely. Software Modularisation “I want my 892 classes to be properly modularised” (insert your ugliest classes here) Objective Function Min. coupling, Max. cohesion
- 14. Test Data Generation “I want the value of x to make the following predicate true” if(x == 3) {...} Objective Function f(x) = |3 - x| This is concrete: the problem defines the objective function directly and concretely. Software Modularisation “I want my 892 classes to be properly modularised” (insert your ugliest classes here) Objective Function This is surrogate: the objective function is something we hope to correlate with some abstract property. Min. coupling, Max. cohesion
- 15. How do we solve problems with surrogate objectives better?
- 16. TAOism
- 17. TAOism
- 18. TAOism
- 19. TAOism
- 20. TAOism
- 21. TAOism Turing-test As Objective function
- 22. TAO Hypothesis ✤ Surrogate objective functions are better learnt then defined.
- 23. A la Brooks... ✤ Having a predefined representation of the problem (objective function) may not produce an intelligent solution ✤ It is better to use the real world (i.e. the real SE problem) as its own representational model ✤ Learner should be given a complete freedom over from which to learn the objective function
- 24. New Problem Instances Turing Test Historical Best Practice All Observables Learnt Objective
- 25. TAO Challenge
- 26. TAO Challenge ✤ Software engineers moan about realistic empirical evaluation
- 27. TAO Challenge ✤ Software engineers moan about realistic empirical evaluation ✤ We interact with open source community very statically
- 28. TAO Challenge ✤ Software engineers moan about realistic empirical evaluation ✤ We interact with open source community very statically ✤ Why don’t we just commit our AI/SBSE/$#@% generated feature/ bug patch/test case/*#$%@ directly to open source projects?
- 29. TAO Challenge ✤ Software engineers moan about realistic empirical evaluation ✤ We interact with open source community very statically ✤ Why don’t we just commit our AI/SBSE/$#@% generated feature/ bug patch/test case/*#$%@ directly to open source projects? ✤ Will they notice? Will they condemn us, or welcome us?
- 30. Summary ✤ Some SE objective functions are merely surrogates ✤ These are better learnt then defined; humans are the final judges ✤ If we are to use AI techniques to solve SE problems, our goal should be Turing Test