Automated Software Engineering
Download as PPTX, PDF2 likes1,270 views
The document discusses the growing importance of modeling in various domains like science, engineering, and software. It notes that models are used everywhere from emergency response to financial decisions. The document argues that automated software engineering techniques are needed to best reason about complex models. It provides several examples of how optimization and evolutionary algorithms have been applied to problems in software engineering like program repair, product line configuration, spacecraft design, and avionics requirements engineering. The overall message is that modeling is a key emerging area and automated approaches are crucial for dealing with large, complex models.
![More lightweight
requirements notations
• E.g. Kang’s product lines
[Kang’90]
• Add in known constraints
– E.g. “if we use a camera
then we need a high
resolution screen”.
• Extract products
– Find subsets of the
product lines that satisfy
constraints.
– If no constraints, linear
time
– Otherwise, can defeat
state-of-the-art optimizers
[Pohl et at, ASE’11]
[Sayyad, Menzies ICSE’13].
Cross-Tree
Constraints
14
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-14-320.jpg)
![Automated software engineering
and optimization
• Many SE activities are like optimization
problems [Harman,Jones’01].
• Due to computational complexity, exact optimization methods
can be impractical for large SBSE problems
• So researchers and practitioners use metaheuristic search to
find near optimal or good-enough solutions.
– E.g. simulated annealing [Rosenbluth et al.’53]
– E.g. genetic algorithms [Goldberg’79]
– E.g. tabu search [Glover86]
15](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-15-320.jpg)
![Optimization and
evolutionary algorithms
• Repeat till happy or exhausted
– Selection (cull the herd)
– Cross-over (the rude bit)
– Mutation (stochastic jiggle)
1
-- better on some
criteria, worse on none
-- generation[i+1] comes
from Pareto frontier of
generation[i]
2
3
5
4
Pareto frontier
7
6
Selection:
8
9
16](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-16-320.jpg)
![More lightweight
requirements notations
• E.g. Kang’s product lines
[Kang’90]
• Add in known constraints
– E.g. “if we use a camera
then we need a high
resolution screen”.
• Extract products
– Find subsets of the
product lines that satisfy
constraints.
– If no constraints, linear
time
– Otherwise, can defeat
state-of-the-art optimizers
[Pohl et at, ASE’11]
[Sayyad, Menzies ICSE’13].
Cross-Tree
Constraints
26
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-26-320.jpg)
![Issues of scale up
• This model: 10 features, 8 rules
• [www.splot-research.org]:
ESHOP: 290 Features, 421 Rules
• LINUX kernel variability project
LINUX x86 kernel
6,888 Features; 344,000 Rules
Cross-Tree Constraints
28
A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the
Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA.
November 2013.](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-28-320.jpg)
![Skip re-entry
• My optimizers vs state of the art numeric optimizers
• My tools: ran 40 times faster
– Generated better solutions
• Powerful succinct explanation tool
31
Automatically Finding the Control Variables for Complex System Behavior Gregory Gay, Tim Menzies, Misty
Davies, and Karen Gundy-Burlet Journal - Automated Software Engineering, 2010 [PDF]](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-31-320.jpg)
![WMC: GIT’s Work Models that
Compute [Kim’11]
• Cognitive models of the agents
(both pilots and computers)
– Late descent,
– Unpredicted rerouting,
– Different tailwind conditions
• Goal: validate operations
procedures (are they safe?)
• NASA’s analysts want to
explore 7000 scenarios.
– With current tools (NSGA-II)
– 300 weeks to complete
• Limited access to hardware
– Queue of researchers wanting
hardware access
– Hardware pulled away if in-flight
incidents for manned
space missions
Asiana Airlines
Flight 214
34
Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim
Menzies, Misty Davies, IEEE Trans Human Factors, to appear](https://image.slidesharecdn.com/ase-141120235239-conversion-gate02/85/Automated-Software-Engineering-34-320.jpg)
Automated Software Engineering
- 1. Automated Software Engineering Cs791 Fall 2015 tjmenzie@ncsu.edu Nov 21, 2014
- 2. Career advice Learn Modeling (it’s the next big thing in SE) 2
- 3. This kind of modeling? No 3
- 4. This kind of modeling? Nope (though it does look pretty cool, heh?) 4
- 6. Software models: are everywhere • If you call an ambulance in London or New York, – those ambulances are controlled by emergency response models. • If you cross the border Arizona to Mexico, – A models determines if you are taken away for extra security measures. • If you default on your car loans, – A model determines when (or if) someone to repossess your car. • If the stock market crashes, – it might be that some model caused the crash. 6
- 7. Software models: are everywhere • Karplus and Levitt – 2013 Nobel prize in chemistry – development of multi-scale models for complex chemical systems – Explored complex chemical reactions (e.g. split-second changes of photosynthesis). 7 • Models are now a central tool in scientific research. – in physics, biology and other fields of science – complex simulations using supercomputers. • E.g. genomic map required analyzing 80 trillion bytes • E.g.. Other computational modeling projects – the rise and fall of native cultures, – subnuclear particles – the Big Bang.
- 8. Question: How to best reason about models? Answer: automated software engineering 8
- 9. The Human Condition: a balancing act between goals • The human condition – a constant balancing act between vaguely understood, often competing, goals. • For example: – Lets build the software faster... – ... with few bugs ... – ... using less budget. • For the consequences of "better, faster, cheaper” : – NASA's loss of the Mars Polar Orbiter – $300 million , wasted 9
- 10. Example • Choices explored by satellite designers at NASA's Jet Propulsion Laboratory. – Dozens of experts in propulsion, communications, guidance and control etc – Week-long sessions – Thrash out possibilities for NASA's next deep space mission. • Groups sitting “here” can make decisions that impact “there”, and they just do not realize. 10
- 11. Keeping track of all those decisions: Lightweight requirements notations • E.g. Martin Feather’s DDP language – Mitigations … – … that retire risks…. – …. That damage requirements • Multi-objective optimization – Find the cheapest mitigations that enable the most requirements 11
- 12. Keeping track of all those decisions: Lightweight requirements notations • My optimizers out-performing state of the art tools – Faster – Simpler policies 12 Data Mining for Very Busy People, Tim Menzies, Ying Hu, IEEE Computer, October 2003
- 13. Keeping track of all those decisions: Lightweight requirements notations • E.g. John Mylopoulos‘ soft goals • Competing requirements • Simulations to explore all the what-ifs Chiang, Eliza, Menzies, Tim, Simulations for very early lifecycle quality evaluations, 13 Software Process: Improvement and Practice 7(3-4), 2003,
- 14. More lightweight requirements notations • E.g. Kang’s product lines [Kang’90] • Add in known constraints – E.g. “if we use a camera then we need a high resolution screen”. • Extract products – Find subsets of the product lines that satisfy constraints. – If no constraints, linear time – Otherwise, can defeat state-of-the-art optimizers [Pohl et at, ASE’11] [Sayyad, Menzies ICSE’13]. Cross-Tree Constraints 14 A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.
- 15. Automated software engineering and optimization • Many SE activities are like optimization problems [Harman,Jones’01]. • Due to computational complexity, exact optimization methods can be impractical for large SBSE problems • So researchers and practitioners use metaheuristic search to find near optimal or good-enough solutions. – E.g. simulated annealing [Rosenbluth et al.’53] – E.g. genetic algorithms [Goldberg’79] – E.g. tabu search [Glover86] 15
- 16. Optimization and evolutionary algorithms • Repeat till happy or exhausted – Selection (cull the herd) – Cross-over (the rude bit) – Mutation (stochastic jiggle) 1 -- better on some criteria, worse on none -- generation[i+1] comes from Pareto frontier of generation[i] 2 3 5 4 Pareto frontier 7 6 Selection: 8 9 16
- 17. Long tradition of software, simulation, and optimization • Not one “best” solutions – Its all trade-offs – Satisficing = satisfy + sacrifice • Problems have to be explored via computer simulations – Try it out and see – And approach pioneered by John Von Neumann, in the 1950s 17
- 18. Explosive growth of SE + optimization Q: Why? A: Thanks to Big Data, more access to more cpu. 18
- 19. Applications of Optimization in SE 1. Requirements Menzies, Feather, Bagnall, Mansouri, Zhang 2. Transformation Cooper, Ryan, Schielke, Subramanian, Fatiregun, Williams 3.Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd 4. Management Alba, Antoniol, Chicano, Di Pentam Greer, Ruhe 5. Heap allocation Cohen, Kooi, Srisa-an 6. Regression test Li, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer 7. SOA Canfora, Di Penta, Esposito, Villani 8. Refactoring Antoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt 9. Test Generation Alba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones, Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao, Wegener, Wilkins 10. Maintenance Antoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift 11. Model checking Alba, Chicano, Godefroid 12. Probing Cohen, Elbaum 13. UIOs Derderian, Guo, Hierons 14. Comprehension Gold, Li, Mahdavi 15. Protocols Alba, Clark, Jacob, Troya 16. Component sel Baker, Skaliotis, Steinhofel, Yoo 17. Agent Oriented Haas, Peysakhov, Sinclair, Shami, Mancoridis 19
- 20. Example 1 (of four) Automated program repair 20
- 21. Multi-objective Optimization, in the 21st century, automated repair A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12 21
- 22. A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12 22
- 23. A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12 23
- 24. • A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12 24
- 25. Example 2 (of four) Software Product Lines 25
- 26. More lightweight requirements notations • E.g. Kang’s product lines [Kang’90] • Add in known constraints – E.g. “if we use a camera then we need a high resolution screen”. • Extract products – Find subsets of the product lines that satisfy constraints. – If no constraints, linear time – Otherwise, can defeat state-of-the-art optimizers [Pohl et at, ASE’11] [Sayyad, Menzies ICSE’13]. Cross-Tree Constraints 26 A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.
- 27. Problem: many competing goals 2 or 3 or 4 or 5 goals Software engineering = navigating the user goals: 1. Satisfy the most domain constraints (0 ≤ #violations ≤ 100%) 2. Offers most features 3. Build “stuff” In least time 4. That we have used most before 5. Using features with least known defects 27 A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.
- 28. Issues of scale up • This model: 10 features, 8 rules • [www.splot-research.org]: ESHOP: 290 Features, 421 Rules • LINUX kernel variability project LINUX x86 kernel 6,888 Features; 344,000 Rules Cross-Tree Constraints 28 A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.
- 29. State of the Art (as of Nov’13) Features 6888 SPLOT Linux (LVAT) 544 290 9 Henard ‘12 White ‘07, ‘08, 09a, 09b, Pohl ‘11 Lopez- Herrejon ‘11 Sayyad, Menzies’13b Sayyad, Menzies’ 13a Velazco ‘13 Johansen ‘11 Benavides ‘05 Shi ‘10, Guo ‘11 Objectives Single-goal Multi-goal 300,000+ clauses 29 A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.
- 30. Example 3 (of four) Space ships! 30
- 31. Skip re-entry • My optimizers vs state of the art numeric optimizers • My tools: ran 40 times faster – Generated better solutions • Powerful succinct explanation tool 31 Automatically Finding the Control Variables for Complex System Behavior Gregory Gay, Tim Menzies, Misty Davies, and Karen Gundy-Burlet Journal - Automated Software Engineering, 2010 [PDF]
- 32. Example 4 (of four) Avionics Requirements Engineering 32
- 33. 33 Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear
- 34. WMC: GIT’s Work Models that Compute [Kim’11] • Cognitive models of the agents (both pilots and computers) – Late descent, – Unpredicted rerouting, – Different tailwind conditions • Goal: validate operations procedures (are they safe?) • NASA’s analysts want to explore 7000 scenarios. – With current tools (NSGA-II) – 300 weeks to complete • Limited access to hardware – Queue of researchers wanting hardware access – Hardware pulled away if in-flight incidents for manned space missions Asiana Airlines Flight 214 34 Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear
- 35. Active Learning: Skip similar examples, focus on the most different 4 mins (GALE) vs 7 hours (rest) Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear
- 36. Career advice Learn Modeling (it’s the next big thing in SE) 36
- 37. Question: How to reason about models? Answer: automated software engineering Cs791, fall 2015 37