SlideShare a Scribd company logo

Intelligent Software Engineering: Synergy between AI and Software Engineering (UCI ISR 2018)

Tao Xie, profile picture
Tao Xie

The document discusses the integration of artificial intelligence (AI) in software engineering, exploring various automated testing tools and methodologies developed through collaborations with industry leaders like Microsoft and Tencent. It highlights advancements in software analytics and intelligent testing tools, including techniques for deep learning system testing and neural machine translation evaluation. Furthermore, it addresses challenges and ethical considerations surrounding AI deployment in real-world applications.

Software
1 of 46
Downloaded 32 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
Intelligent Software Engineering: Synergy between AI and Software Engineering Tao Xie University of Illinois at Urbana-Champaign taoxie@illinois.edu http://taoxie.cs.illinois.edu/
Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
https://techcrunch.com/2016/08/05/carnegie-mellons-mayhem-ai-takes-home-2-million-from-darpas-cyber-grand-challenge/
Dynamic Symbolic Execution Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123
}a==null a.Length>0 a[0]==123
 T TF T F F Execute&MonitorSolve Choose next path Done: There is no path left. Negated condition [DART: Godefroid et al. PLDI’05] Z3Constraint solver has decision procedures for - Arrays - Linear integer arithmetic - Bitvector arithmetic - Floating-point arithmetic - 

Past: Automated Software Testing ‱ 10 years of collaboration with Microsoft Research on Pex [ASE’14 Ex] ‱ .NET Test Generation Tool based on Dynamic Symbolic Execution ‱ Tackle challenges of ‱ Path explosion via fitness function [DSN’09] ‱ Method sequence explosion via program synthesis [OOPSLA’11] ‱ 
 ‱ Shipped in Visual Studio 2015/2017 Enterprise Edition ‱ As IntelliTest Tillmann, de Halleux, Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. ASE’14 Experience Papers http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
Past: Android App Testing ‱ 2 years of collaboration with Tencent Inc. WeChat testing team ‱ Guided Random Test Generation Tool improved over Google Monkey ‱ Resulting tool deployed in daily WeChat testing practice ‱ WeChat = WhatsApp + Facebook + Instagram + PayPal + Uber 
 ‱ #monthly active users: 1 billion @2018 March ‱ Daily#: dozens of billion messages sent, hundreds of million photos uploaded, hundreds of million payment transactions executed ‱ First studies on testing industrial Android apps [FSE’16IN][ICSE’17SEIP] ‱ Beyond open source Android apps focused by academia WeChat
Intelligent Android App Testing @ Facebook https://code.fb.com/developer-tools/sapienz-intelligent-automated-software-testing-at-scale/ https://www.forbes.com/sites/parmyolson/2018/11/29/facebook-has-a-bug-hunting-tool-that-mimics-human-behavior/
Android Test Generation Tools: A Retrospective Monkey Official Blind random 9 2008 2017 Stoat FSE ’17 Model-based evolutionary 2012 GUIRipper ASE ‘12 Model-based 2013 2015 2016... .. A3E OOPSLA ‘13 Systematic Dynodroid FSE ‘13 Guided random SwiftHand OOPSLA ‘13 Model-based DroidBot ICSE-C ‘17 Model-based ACTEve FSE ‘12 Concolic WCTester FSE-Ind ‘16 Guided random Sapienz ISSTA ‘16 Evolutionary Study by Choudhary et al. ASE ‘15 How do these tools perform on industrial apps that people actually use everyday?
Android Test Generation Tools: Existing Evaluations 10 2017 Stoat FSE ’17 Model-based evolutionary 2012 GUIRipper ASE ‘12 Model-based 2013 2015 2016.. A3E OOPSLA ‘13 Systematic Dynodroid FSE ‘13 Guided random SwiftHand OOPSLA ‘13 Model-based DroidBot ICSE-C ‘17 Model-based ACTEve FSE ‘12 Concolic WCTester FSE-Ind ‘16 Guided random Sapienz ISSTA ‘16 Evolutionary Study by Choudhary et al. ASE ‘15 Industrial apps not involved Industrial apps limitedly involved Single case study only There is no comprehensive comparison among existing tools over industrial apps.  Does a newly proposed tool really outperform existing tools (especially Monkey) on industrial apps? Wang, Li, Yang, Cao, Zhang, Deng, Xie. An Empirical Study of Android Test Generation Tools in Industrial Cases. ASE’18. http://taoxie.cs.illinois.edu/publications/ase18-androidtest.pdf
Next: Intelligent Software Testing ‱ Learning from others working on the same things ‱ Our work on mining API usage method sequences to test the API [ESEC/FSE’09: MSeqGen] ‱ Visser et al. Green: Reducing, reusing and recycling constraints in program analysis. [FSE’12] ‱ Learning from others working on similar things ‱ Jia et al. Enhancing reuse of constraint solutions to improve symbolic execution. [ISSTA’15] ‱ Aquino et al. Heuristically Matching Solution Spaces of Arithmetic Formulas to Efficiently Reuse Solutions. [ICSE’17] [Jia et al. ISSTA’15] Continuous Learning
Software Analytics Software analytics is to enable software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information for data- driven tasks around software and services. Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as a Learning Case in Practice: Approaches and Experiences. In MALETS 2011 http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf
Software Analytics Software analytics is to enable software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information for data- driven tasks around software and services. Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as a Learning Case in Practice: Approaches and Experiences. In MALETS 2011 http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf
Data sources Runtime traces Program logs System events Perf counters 
 Usage log User surveys Online forum posts Blog & Twitter 
 Source code Bug history Check-in history Test cases Eye tracking MRI/EMG 

Research Topics & Technology Pillars
Past: Software Analytics ‱ StackMine [ICSE’12, IEEESoft’13]: performance debugging in the large ‱ Data Source: Performance call stack traces from Windows end users ‱ Analytics Output: Ranked clusters of call stack traces based on shared patterns ‱ Impact: Deployed/used in daily practice of Windows Performance Analysis team ‱ XIAO [ACSAC’12, ICSE’17 SEIP]: code-clone detection and search ‱ Data Source: Source code repos (+ given code segment optionally) ‱ Analytics Output: Code clones ‱ Impact: Shipped in Visual Studio 2012; deployed/used in daily practice of Microsoft Security Response Center Internet @Microsoft Research Asia
Past: Software Analytics ‱ Service Analysis Studio [ASE’13-EX]: service incident management ‱ Data Source: Transaction logs, system metrics, past incident reports ‱ Analytics Output: Healing suggestions/likely root causes of the given incident ‱ Impact: Deployed and used by an important Microsoft service (hundreds of millions of users) for incident management @Microsoft Research Asia
Open Source Microservice Benchmark System TrainTicket 70+ microservices, including 41 business ones, 30 infrastructure ones (message middleware service, distributed cache services, database services), totally 300K LOC Git Repohttps://github.com/microcosmx/train_ticket ‱ Include Java、Python、Go、 Node.js ‱ Use asynchronous communication and queue ‱ Substantial test cases including 100+ unit and integration tests ‱ Visualization tools for runtime monitoring and management Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chenjie Xu, Chao Ji, and Wenyun Zhao. Poster: Benchmarking Microservice Systems for Software Engineering Research. ICSE 2018 Posters. http://taoxie.cs.illinois.edu/publications/icse18poster-microservices.pdf Fudan、UIUC、SUTD Collaborative Research
Current: Modern IDE
Next: Intelligent IDE https://www.hksilicon.com/articles/1213020 Inspired by Software Analytics GroupNatural language interfacing
Translation of NL to Regular Expressions/SQL ● Program Aliasing: a semantically equivalent program may have many syntactically different forms NL Sentences
NL  Regex: sequence-to-sequence model ● Encoder/Decoder: 2 layers stacked LSTM architectures [Locascio et al. EMNLP’16]
Training Objective: Maximum Likelihood Estimation (MLE)  Maximizing Semantic Correctness ● Standard seq-to-seq maximizes likelihood mapping NL to ground truth ● MLE penalizes syntactically different but semantically equivalent regex ● Reward : semantic correctness ● Alternative objective: Maximize the expected Leveraging the REINFORCE technique of policy gradient [William’92] to maximize Expected Semantic Correctness Zhong, Guo, Yang, Peng, Xie, Lou, Liu, Zhang. SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications. EMNLP’18. http://taoxie.cs.illinois.edu/publications/emnlp18-semregex.pdf
Measurements of Semantic Correctness ([ABab]&[A-Z]).*X● Minimal DFAs ● Test Cases (pos/neg string examples)
Evaluation Results of NLRegex Approaches Zhong, Guo, Yang, Peng, Xie, Lou, Liu, Zhang. SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications. EMNLP’18. http://taoxie.cs.illinois.edu/publications/emnlp18-semregex.pdf DFA-equivalence Accuracy
https://medium.com/ai-for-software-engineering/ai-for-software-engineering-industry-landscape-12-aug-2018-e8e028628663 Intelligent SEIndustry
Intelligent SETech Transfer http://www.diffblue.com/ Oxford University spin-off, Daniel Kroening et al. Peking University spin- off, Ge Li et al. https://www.codota.com/ Technion spin-off, Eran Yahav et al. Technical University Munich spin-off, Benedikt Hauptmann et al. https://www.qualicen.de/en/ http://www.aixcoder.com/ MaJiCKe UCL spin-off, Mark Harman et al. Acquired by Facebook http://www.engineering.ucl.ac.uk/news/bug-finding-majicke-finds-home-facebook/ https://www.deepcode.ai/ ETH Zurich spin-off, Martin Vechev et al.
Quite Many Recent Papers in AI/ML for SE https://ml4code.github.io/ ‱ 2018 (35) ‱ 2017 (34) ‱ 2016 (25) ‱ 2015 (25) ‱ 2014 (14) ‱ 2013 (9) ‱ 2012 (1) ‱ 2009 (1) ‱ 2007 (1)https://arxiv.org/abs/1709.06182
Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
White-House-Sponsored Workshop (2016 June 28) http://www.cmu.edu/safartint/
Self-Driving Tesla Involved in Fatal Crash (2016 June 30) http://www.nytimes.com/2016/07/01/business/self-driving-tesla-fatal-crash-investigation.html “A Tesla car in autopilot crashed into a trailer because the autopilot system failed to recognize the trailer as an obstacle due to its “white color against a brightly lit sky” and the “high ride height” http://www.cs.columbia.edu/~suman/docs/deepxplore.pdf
(March 18, 2018) http://fortune.com/2018/03/19/uber-halts-self-driving-car-testing-fatal-accident-tempe-a https://www.theguardian.com/technology/2018/aug/29/coding-algorithms-frankenalgos-program-danger
Microsoft's Teen Chatbot Tay Turned into Genocidal Racist (2016 March 23/24) http://www.businessinsider.com/ai-expert-explains-why-microsofts-tay-chatbot-is-so-racist-2016-3 "There are a number of precautionary steps they [Microsoft] could have taken. It wouldn't have been too hard to create a blacklist of terms; or narrow the scope of replies. They could also have simply manually moderated Tay for the first few days, even if that had meant slower responses." “businesses and other AI developers will need to give more thought to the protocols they design for testing and training AIs like Tay.”
Amazon’s AI Tool Discriminating Against Women https://www.businessinsider.com/amazon-built-ai-to-hire-people-discriminated-against-women-2018-10
Amazon’s AI Tool Discriminating Against Women https://www.businessinsider.com/amazon-built-ai-to-hire-people-discriminated-against-women-2018-10
Adversarial Machine Learning/Testing ● Adversarial testing [Szegedy et al. ICLR’14]: find corner-case inputs imperceptible to human but induce errors 36 School bus OstrichCarefully crafted noise Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Slide adapted from SOSP’17 slides
DeepXplore: Automated Whitebox Testing of Deep Learning Systems ‱ Systematic testing of Deep Neural Nets (DNNs) ‱ Neuron coverage: testing coverage metric for deep nerual net ‱ Automated: cross-check multiple DNNs ‱ Realistic: physically realizable transformations (e.g., lighting) ‱ Effective: ‱ 15 State-of-the-art DNNs on 5 large datasets (ImageNet, Self-driving cars, PDF/Android malware) ‱ Numerous corner-case errors ‱ 50% more neuron coverage than existing testing No accident Darker: Accident DeepXplore 37 Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Slide adapted from SOSP’17 slides
Example Detected Erroneous Behaviors Turn rightGo straight 38 Go straight Turn left Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE 2018. Slide adapted from SOSP’17 slides
Example Detected Erroneous Behaviors Turn rightGo straight 39 Go straight Turn left Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE 2018. Slide adapted from SOSP’17 slides Lu et al. NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles. CVPR’17. https://arxiv.org/abs/1707.03501“ “
Neural Machine Translation Screen snapshot captured on April 5, 2018 ‱ Overall better than statistical machine translation ‱ Worse controllability ‱ Existing translation quality assurance  Need reference translationnot applicable online  Cannot precisely locate problem types and
Translation Quality Assurance ● Key ideablack-box algorithms specialized for common problems ○ No need for reference translation; need only the original sentence and generated translation ○ Precise problem localization ● Common problems ○ Under-translation ○ Over-translation Tencent、UIUC Collaborative Work Zheng, Wang, Liu, Zhang, Zeng, Deng, Yang, He, Xie. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv:1807.02340, July 2018.
Industry Impact ● Adopted to improve WeChat translation service (over 1 billion users online serving 12 million translation tasks) ○ Offline monitoring (regression testing) ○ Online monitoring (real time selection of best model) ● Large scale test data for translation ○ ~130K English/180K Chinese words/phrases ○ Detect numerous problems in Google Translate and YouDao BLEU Score Improvement %Problems Reduction Problem Cases in Other Translation Services Tencent、UIUC Collaborative Work Zheng, Wang, Liu, Zhang, Zeng, Deng, Yang, He, Xie. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv:1807.02340, July 2018.
Quite Many Recent Papers in SE for AI/ML ‱ Ma et al. MODE: Automated Neural Network Model Debugging via State Differential Analysis and Input Selection. ESEC/FSE’18 ‱ Sun et al. Concolic Testing for Deep Neural Networks. ASE’18 ‱ Udeshi et al. Automated Directed Fairness Testing. ASE’18 ‱ Ma et al. DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems. ASE’18 ‱ Zhang et al. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. ASE’18 ‱ Dwarakanath et al. Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing. ISSTA’18 ‱ Zhang et al. An Empirical Study on TensorFlow Program Bugs. ISSTA’18 ‱ Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE’18 ‱ Abdessalem et al. Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms. ICSE’18 ‱ Odena, Goodfellow. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. arXiv:1807.10875. 2018. ‱ 

Trustworthy Machine Learning https://news.psu.edu/story/543643/2018/10/24/research/nsf-selects-penn-state-create-lead-center-trustworthy-machine
(SE  AI)  Practice Impact Problem Domain Solution Domain Practice Intelligent Software Engineering Intelligence Software Engineering
46 Thank You! Q & A This work was supported in part by NSF under grants no. CNS-1513939, CNS-1564274, CCF-1816615, and a grant from the ZJUI Research Program.

More Related Content

PDF
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
Tao Xie
 
PDF
SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So...
Tao Xie
 
PDF
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
Tao Xie
 
PPTX
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...
Tao Xie
 
PPTX
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie
 
PDF
Software Analytics: Data Analytics for Software Engineering and Security
Tao Xie
 
PPTX
ACM Chicago March 2019 meeting: Software Engineering and AI - Prof. Tao Xie, ...
ACM Chicago
 
PDF
ISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven Research
Tao Xie
 
MSRA 2018: Intelligent Software Engineering: Synergy between AI and Software ...
Tao Xie
 
SETTA'18 Keynote: Intelligent Software Engineering: Synergy between AI and So...
Tao Xie
 
ISEC'18 Keynote: Intelligent Software Engineering: Synergy between AI and Sof...
Tao Xie
 
DSML 2021 Keynote: Intelligent Software Engineering: Working at the Intersect...
Tao Xie
 
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie
 
Software Analytics: Data Analytics for Software Engineering and Security
Tao Xie
 
ACM Chicago March 2019 meeting: Software Engineering and AI - Prof. Tao Xie, ...
ACM Chicago
 
ISEC'18 Tutorial: Research Methodology on Pursuing Impact-Driven Research
Tao Xie
 

What's hot (20)

PDF
Planning and Executing Practice-Impactful Research
Tao Xie
 
PDF
Software bug prediction
Muthukumaran Kasinathan
 
PDF
Measuring Agile Software Development
Miroslaw Staron
 
PDF
AI for Software Engineering
Miroslaw Staron
 
PDF
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
Trivadis
 
PPTX
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Patrizio Pelliccione
 
PDF
PhD Proposal talk
Ray Buse
 
PPTX
Self Adaptive Systems
Adeel Rasheed
 
PDF
Towards Fairer Datasets: Filtering and Balancing the Distribution of the Peop...
QuantUniversity
 
PDF
Se research update
Nacha Chondamrongkul
 
PDF
Aakiel abernathy resume
Aakiel Abernathy
 
PDF
Publish or Perish: Questioning the Impact of Our Research on the Software Dev...
Margaret-Anne Storey
 
PDF
Opinion Mining for Software Engineering
Alexander Serebrenik
 
PDF
Put Your Hands in the Mud: What Technique, Why, and How
Massimiliano Di Penta
 
PDF
Implications of Open Source Software Use (or Let's Talk Open Source)
Gail Murphy
 
PDF
A Multi-Objective Refactoring Approach to Introduce Design Patterns and Fix A...
Ali Ouni
 
PDF
Empirical evaluation in 2020: how big, how beautiful?
Massimiliano Di Penta
 
PDF
Analyzing Big Data's Weakest Link (hint: it might be you)
HPCC Systems
 
PDF
Summarization Techniques for Code, Change, Testing and User Feedback
Sebastiano Panichella
 
DOCX
Carol Harstad Research Proposal
Carol Harstad
 
Planning and Executing Practice-Impactful Research
Tao Xie
 
Software bug prediction
Muthukumaran Kasinathan
 
Measuring Agile Software Development
Miroslaw Staron
 
AI for Software Engineering
Miroslaw Staron
 
TechEvent 2019: Artificial Intelligence in Dev & Ops; Martin Luckow - Trivadis
Trivadis
 
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Patrizio Pelliccione
 
PhD Proposal talk
Ray Buse
 
Self Adaptive Systems
Adeel Rasheed
 
Towards Fairer Datasets: Filtering and Balancing the Distribution of the Peop...
QuantUniversity
 
Se research update
Nacha Chondamrongkul
 
Aakiel abernathy resume
Aakiel Abernathy
 
Publish or Perish: Questioning the Impact of Our Research on the Software Dev...
Margaret-Anne Storey
 
Opinion Mining for Software Engineering
Alexander Serebrenik
 
Put Your Hands in the Mud: What Technique, Why, and How
Massimiliano Di Penta
 
Implications of Open Source Software Use (or Let's Talk Open Source)
Gail Murphy
 
A Multi-Objective Refactoring Approach to Introduce Design Patterns and Fix A...
Ali Ouni
 
Empirical evaluation in 2020: how big, how beautiful?
Massimiliano Di Penta
 
Analyzing Big Data's Weakest Link (hint: it might be you)
HPCC Systems
 
Summarization Techniques for Code, Change, Testing and User Feedback
Sebastiano Panichella
 
Carol Harstad Research Proposal
Carol Harstad
 
Ad

Similar to Intelligent Software Engineering: Synergy between AI and Software Engineering (UCI ISR 2018) (20)

PPTX
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie
 
PPTX
Impact-Driven Research on Software Engineering Tooling
Tao Xie
 
PDF
01 - Course setup software sustainability
Seyidali
 
PDF
Applying AI to software engineering problems: Do not forget the human!
University of CĂłrdoba
 
PPTX
Bertrand Meyer - Challenges in computing research at SIT Insights in Technolo...
Schaffhausen Institute of Technology
 
PDF
Software Mining and Software Datasets
Tao Xie
 
PPT
SE and AI: a two-way street
CS, NcState
 
PDF
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...
gerogepatton
 
PDF
SOFTWARE TESTING: ISSUES AND CHALLENGES OF ARTIFICIAL INTELLIGENCE & MACHINE ...
ijaia
 
PDF
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...
gerogepatton
 
PDF
November 2024 - Top 10 Read Articles in Software Engineering & Applications
sebastianku31
 
PDF
Artificial Intelligence for Automated Software Testing
Lionel Briand
 
PPTX
Software Engineering Challenges in building AI-based complex systems
Ivica Crnkovic
 
PDF
Software Testing:‹ A Research Travelogue ‹(2000–2014)
Alex Orso
 
PDF
AI Based Testing - A Comprehensive Guide.pdf
kalichargn70th171
 
PDF
Rise of the Machines: Can Artificial Intelligence Terminate Manual Testing?
TechWell
 
PDF
AI for Software Testing Excellence in 2024
Testgrid.io
 
PDF
Integrating AI in software quality in absence of a well-defined requirements
Nagarro
 
PPTX
ARTIFICIAL INTELLIGENCE IN SOFTWARE ENGINEERING
SahithiKotha2
 
PDF
Software testing
Enamul Haque
 
Intelligent Software Engineering: Synergy between AI and Software Engineering
Tao Xie
 
Impact-Driven Research on Software Engineering Tooling
Tao Xie
 
01 - Course setup software sustainability
Seyidali
 
Applying AI to software engineering problems: Do not forget the human!
University of CĂłrdoba
 
Bertrand Meyer - Challenges in computing research at SIT Insights in Technolo...
Schaffhausen Institute of Technology
 
Software Mining and Software Datasets
Tao Xie
 
SE and AI: a two-way street
CS, NcState
 
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...
gerogepatton
 
SOFTWARE TESTING: ISSUES AND CHALLENGES OF ARTIFICIAL INTELLIGENCE & MACHINE ...
ijaia
 
Software Testing: Issues and Challenges of Artificial Intelligence & Machine ...
gerogepatton
 
November 2024 - Top 10 Read Articles in Software Engineering & Applications
sebastianku31
 
Artificial Intelligence for Automated Software Testing
Lionel Briand
 
Software Engineering Challenges in building AI-based complex systems
Ivica Crnkovic
 
Software Testing:‹ A Research Travelogue ‹(2000–2014)
Alex Orso
 
AI Based Testing - A Comprehensive Guide.pdf
kalichargn70th171
 
Rise of the Machines: Can Artificial Intelligence Terminate Manual Testing?
TechWell
 
AI for Software Testing Excellence in 2024
Testgrid.io
 
Integrating AI in software quality in absence of a well-defined requirements
Nagarro
 
ARTIFICIAL INTELLIGENCE IN SOFTWARE ENGINEERING
SahithiKotha2
 
Software testing
Enamul Haque
 
Ad

More from Tao Xie (19)

PDF
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...
Tao Xie
 
PDF
Diversity and Computing/Engineering: Perspectives from Allies
Tao Xie
 
PDF
Software Analytics: Data Analytics for Software Engineering
Tao Xie
 
PDF
Transferring Software Testing Tools to Practice (AST 2017 Keynote)
Tao Xie
 
PPTX
Transferring Software Testing Tools to Practice
Tao Xie
 
PPTX
Advances in Unit Testing: Theory and Practice
Tao Xie
 
PDF
Common Technical Writing Issues
Tao Xie
 
PPTX
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck
Tao Xie
 
PPTX
Transferring Software Testing and Analytics Tools to Practice
Tao Xie
 
PDF
User Expectations in Mobile App Security
Tao Xie
 
PDF
Software Analytics - Achievements and Challenges
Tao Xie
 
PPTX
Next Generation Developer Testing: Parameterized Testing
Tao Xie
 
PPTX
Csise15 codehunt
Tao Xie
 
PDF
Text Analytics for Security
Tao Xie
 
PPTX
Gamifying Teaching and Learning of Software Engineering and Programming
Tao Xie
 
PPTX
Towards Mining Software Repositories Research that Matters
Tao Xie
 
PDF
Tutorial: Text Analytics for Security
Tao Xie
 
PPTX
Software Analytics: Towards Software Mining that Matters (2014)
Tao Xie
 
PPTX
Teaching and Learning Programming and Software Engineering via Interactive Ga...
Tao Xie
 
MSR 2022 Foundational Contribution Award Talk: Software Analytics: Reflection...
Tao Xie
 
Diversity and Computing/Engineering: Perspectives from Allies
Tao Xie
 
Software Analytics: Data Analytics for Software Engineering
Tao Xie
 
Transferring Software Testing Tools to Practice (AST 2017 Keynote)
Tao Xie
 
Transferring Software Testing Tools to Practice
Tao Xie
 
Advances in Unit Testing: Theory and Practice
Tao Xie
 
Common Technical Writing Issues
Tao Xie
 
HotSoS16 Tutorial "Text Analytics for Security" by Tao Xie and William Enck
Tao Xie
 
Transferring Software Testing and Analytics Tools to Practice
Tao Xie
 
User Expectations in Mobile App Security
Tao Xie
 
Software Analytics - Achievements and Challenges
Tao Xie
 
Next Generation Developer Testing: Parameterized Testing
Tao Xie
 
Csise15 codehunt
Tao Xie
 
Text Analytics for Security
Tao Xie
 
Gamifying Teaching and Learning of Software Engineering and Programming
Tao Xie
 
Towards Mining Software Repositories Research that Matters
Tao Xie
 
Tutorial: Text Analytics for Security
Tao Xie
 
Software Analytics: Towards Software Mining that Matters (2014)
Tao Xie
 
Teaching and Learning Programming and Software Engineering via Interactive Ga...
Tao Xie
 

Recently uploaded (20)

PDF
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
PDF
top salesforce developer skills in 2025.pdf
CloudMetic
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PPTX
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 
Navsoft: AI-Powered Business Solutions & Custom Software Development
Navsoft
 
top salesforce developer skills in 2025.pdf
CloudMetic
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
CHAPTER 2 - PM Management and IT Context
KevinPutra14
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
lawrenceomai2
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
ManageIQ - Sprint 268 Review - Slide Deck
ManageIQ
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
Transform Your Business with a Software ERP System
Canada Software Company
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
AnujKumar530915
 

Intelligent Software Engineering: Synergy between AI and Software Engineering (UCI ISR 2018)

  • 1. Intelligent Software Engineering: Synergy between AI and Software Engineering Tao Xie University of Illinois at Urbana-Champaign taoxie@illinois.edu http://taoxie.cs.illinois.edu/
  • 2. Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
  • 3. Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
  • 5. Dynamic Symbolic Execution Code to generate inputs for: Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==1234567890 void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } Observed constraints a==null a!=null && !(a.Length>0) a!=null && a.Length>0 && a[0]!=1234567890 a!=null && a.Length>0 && a[0]==1234567890 Data null {} {0} {123
}a==null a.Length>0 a[0]==123
 T TF T F F Execute&MonitorSolve Choose next path Done: There is no path left. Negated condition [DART: Godefroid et al. PLDI’05] Z3Constraint solver has decision procedures for - Arrays - Linear integer arithmetic - Bitvector arithmetic - Floating-point arithmetic - 

  • 6. Past: Automated Software Testing ‱ 10 years of collaboration with Microsoft Research on Pex [ASE’14 Ex] ‱ .NET Test Generation Tool based on Dynamic Symbolic Execution ‱ Tackle challenges of ‱ Path explosion via fitness function [DSN’09] ‱ Method sequence explosion via program synthesis [OOPSLA’11] ‱ 
 ‱ Shipped in Visual Studio 2015/2017 Enterprise Edition ‱ As IntelliTest Tillmann, de Halleux, Xie. Transferring an Automated Test Generation Tool to Practice: From Pex to Fakes and Code Digger. ASE’14 Experience Papers http://taoxie.cs.illinois.edu/publications/ase14-pexexperiences.pdf
  • 7. Past: Android App Testing ‱ 2 years of collaboration with Tencent Inc. WeChat testing team ‱ Guided Random Test Generation Tool improved over Google Monkey ‱ Resulting tool deployed in daily WeChat testing practice ‱ WeChat = WhatsApp + Facebook + Instagram + PayPal + Uber 
 ‱ #monthly active users: 1 billion @2018 March ‱ Daily#: dozens of billion messages sent, hundreds of million photos uploaded, hundreds of million payment transactions executed ‱ First studies on testing industrial Android apps [FSE’16IN][ICSE’17SEIP] ‱ Beyond open source Android apps focused by academia WeChat
  • 8. Intelligent Android App Testing @ Facebook https://code.fb.com/developer-tools/sapienz-intelligent-automated-software-testing-at-scale/ https://www.forbes.com/sites/parmyolson/2018/11/29/facebook-has-a-bug-hunting-tool-that-mimics-human-behavior/
  • 9. Android Test Generation Tools: A Retrospective Monkey Official Blind random 9 2008 2017 Stoat FSE ’17 Model-based evolutionary 2012 GUIRipper ASE ‘12 Model-based 2013 2015 2016... .. A3E OOPSLA ‘13 Systematic Dynodroid FSE ‘13 Guided random SwiftHand OOPSLA ‘13 Model-based DroidBot ICSE-C ‘17 Model-based ACTEve FSE ‘12 Concolic WCTester FSE-Ind ‘16 Guided random Sapienz ISSTA ‘16 Evolutionary Study by Choudhary et al. ASE ‘15 How do these tools perform on industrial apps that people actually use everyday?
  • 10. Android Test Generation Tools: Existing Evaluations 10 2017 Stoat FSE ’17 Model-based evolutionary 2012 GUIRipper ASE ‘12 Model-based 2013 2015 2016.. A3E OOPSLA ‘13 Systematic Dynodroid FSE ‘13 Guided random SwiftHand OOPSLA ‘13 Model-based DroidBot ICSE-C ‘17 Model-based ACTEve FSE ‘12 Concolic WCTester FSE-Ind ‘16 Guided random Sapienz ISSTA ‘16 Evolutionary Study by Choudhary et al. ASE ‘15 Industrial apps not involved Industrial apps limitedly involved Single case study only There is no comprehensive comparison among existing tools over industrial apps.  Does a newly proposed tool really outperform existing tools (especially Monkey) on industrial apps? Wang, Li, Yang, Cao, Zhang, Deng, Xie. An Empirical Study of Android Test Generation Tools in Industrial Cases. ASE’18. http://taoxie.cs.illinois.edu/publications/ase18-androidtest.pdf
  • 11. Next: Intelligent Software Testing ‱ Learning from others working on the same things ‱ Our work on mining API usage method sequences to test the API [ESEC/FSE’09: MSeqGen] ‱ Visser et al. Green: Reducing, reusing and recycling constraints in program analysis. [FSE’12] ‱ Learning from others working on similar things ‱ Jia et al. Enhancing reuse of constraint solutions to improve symbolic execution. [ISSTA’15] ‱ Aquino et al. Heuristically Matching Solution Spaces of Arithmetic Formulas to Efficiently Reuse Solutions. [ICSE’17] [Jia et al. ISSTA’15] Continuous Learning
  • 12. Software Analytics Software analytics is to enable software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information for data- driven tasks around software and services. Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as a Learning Case in Practice: Approaches and Experiences. In MALETS 2011 http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf
  • 13. Software Analytics Software analytics is to enable software practitioners to perform data exploration and analysis in order to obtain insightful and actionable information for data- driven tasks around software and services. Dongmei Zhang, Yingnong Dang, Jian-Guang Lou, Shi Han, Haidong Zhang, and Tao Xie. Software Analytics as a Learning Case in Practice: Approaches and Experiences. In MALETS 2011 http://research.microsoft.com/en-us/groups/sa/malets11-analytics.pdf
  • 14. Data sources Runtime traces Program logs System events Perf counters 
 Usage log User surveys Online forum posts Blog & Twitter 
 Source code Bug history Check-in history Test cases Eye tracking MRI/EMG 

  • 15. Research Topics & Technology Pillars
  • 16. Past: Software Analytics ‱ StackMine [ICSE’12, IEEESoft’13]: performance debugging in the large ‱ Data Source: Performance call stack traces from Windows end users ‱ Analytics Output: Ranked clusters of call stack traces based on shared patterns ‱ Impact: Deployed/used in daily practice of Windows Performance Analysis team ‱ XIAO [ACSAC’12, ICSE’17 SEIP]: code-clone detection and search ‱ Data Source: Source code repos (+ given code segment optionally) ‱ Analytics Output: Code clones ‱ Impact: Shipped in Visual Studio 2012; deployed/used in daily practice of Microsoft Security Response Center Internet @Microsoft Research Asia
  • 17. Past: Software Analytics ‱ Service Analysis Studio [ASE’13-EX]: service incident management ‱ Data Source: Transaction logs, system metrics, past incident reports ‱ Analytics Output: Healing suggestions/likely root causes of the given incident ‱ Impact: Deployed and used by an important Microsoft service (hundreds of millions of users) for incident management @Microsoft Research Asia
  • 18. Open Source Microservice Benchmark System TrainTicket 70+ microservices, including 41 business ones, 30 infrastructure ones (message middleware service, distributed cache services, database services), totally 300K LOC Git Repohttps://github.com/microcosmx/train_ticket ‱ Include Java、Python、Go、 Node.js ‱ Use asynchronous communication and queue ‱ Substantial test cases including 100+ unit and integration tests ‱ Visualization tools for runtime monitoring and management Xiang Zhou, Xin Peng, Tao Xie, Jun Sun, Chenjie Xu, Chao Ji, and Wenyun Zhao. Poster: Benchmarking Microservice Systems for Software Engineering Research. ICSE 2018 Posters. http://taoxie.cs.illinois.edu/publications/icse18poster-microservices.pdf Fudan、UIUC、SUTD Collaborative Research
  • 20. Next: Intelligent IDE https://www.hksilicon.com/articles/1213020 Inspired by Software Analytics GroupNatural language interfacing
  • 21. Translation of NL to Regular Expressions/SQL ● Program Aliasing: a semantically equivalent program may have many syntactically different forms NL Sentences
  • 22. NL  Regex: sequence-to-sequence model ● Encoder/Decoder: 2 layers stacked LSTM architectures [Locascio et al. EMNLP’16]
  • 23. Training Objective: Maximum Likelihood Estimation (MLE)  Maximizing Semantic Correctness ● Standard seq-to-seq maximizes likelihood mapping NL to ground truth ● MLE penalizes syntactically different but semantically equivalent regex ● Reward : semantic correctness ● Alternative objective: Maximize the expected Leveraging the REINFORCE technique of policy gradient [William’92] to maximize Expected Semantic Correctness Zhong, Guo, Yang, Peng, Xie, Lou, Liu, Zhang. SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications. EMNLP’18. http://taoxie.cs.illinois.edu/publications/emnlp18-semregex.pdf
  • 24. Measurements of Semantic Correctness ([ABab]&[A-Z]).*X● Minimal DFAs ● Test Cases (pos/neg string examples)
  • 25. Evaluation Results of NLRegex Approaches Zhong, Guo, Yang, Peng, Xie, Lou, Liu, Zhang. SemRegex: A Semantics-Based Approach for Generating Regular Expressions from Natural Language Specifications. EMNLP’18. http://taoxie.cs.illinois.edu/publications/emnlp18-semregex.pdf DFA-equivalence Accuracy
  • 27. Intelligent SETech Transfer http://www.diffblue.com/ Oxford University spin-off, Daniel Kroening et al. Peking University spin- off, Ge Li et al. https://www.codota.com/ Technion spin-off, Eran Yahav et al. Technical University Munich spin-off, Benedikt Hauptmann et al. https://www.qualicen.de/en/ http://www.aixcoder.com/ MaJiCKe UCL spin-off, Mark Harman et al. Acquired by Facebook http://www.engineering.ucl.ac.uk/news/bug-finding-majicke-finds-home-facebook/ https://www.deepcode.ai/ ETH Zurich spin-off, Martin Vechev et al.
  • 28. Quite Many Recent Papers in AI/ML for SE https://ml4code.github.io/ ‱ 2018 (35) ‱ 2017 (34) ‱ 2016 (25) ‱ 2015 (25) ‱ 2014 (14) ‱ 2013 (9) ‱ 2012 (1) ‱ 2009 (1) ‱ 2007 (1)https://arxiv.org/abs/1709.06182
  • 29. Artificial Intelligence  Software Engineering Artificial Intelligence Software Engineering Intelligent Software Engineering Intelligence Software Engineering
  • 30. White-House-Sponsored Workshop (2016 June 28) http://www.cmu.edu/safartint/
  • 31. Self-Driving Tesla Involved in Fatal Crash (2016 June 30) http://www.nytimes.com/2016/07/01/business/self-driving-tesla-fatal-crash-investigation.html “A Tesla car in autopilot crashed into a trailer because the autopilot system failed to recognize the trailer as an obstacle due to its “white color against a brightly lit sky” and the “high ride height” http://www.cs.columbia.edu/~suman/docs/deepxplore.pdf
  • 32. (March 18, 2018) http://fortune.com/2018/03/19/uber-halts-self-driving-car-testing-fatal-accident-tempe-a https://www.theguardian.com/technology/2018/aug/29/coding-algorithms-frankenalgos-program-danger
  • 33. Microsoft's Teen Chatbot Tay Turned into Genocidal Racist (2016 March 23/24) http://www.businessinsider.com/ai-expert-explains-why-microsofts-tay-chatbot-is-so-racist-2016-3 "There are a number of precautionary steps they [Microsoft] could have taken. It wouldn't have been too hard to create a blacklist of terms; or narrow the scope of replies. They could also have simply manually moderated Tay for the first few days, even if that had meant slower responses." “businesses and other AI developers will need to give more thought to the protocols they design for testing and training AIs like Tay.”
  • 34. Amazon’s AI Tool Discriminating Against Women https://www.businessinsider.com/amazon-built-ai-to-hire-people-discriminated-against-women-2018-10
  • 35. Amazon’s AI Tool Discriminating Against Women https://www.businessinsider.com/amazon-built-ai-to-hire-people-discriminated-against-women-2018-10
  • 36. Adversarial Machine Learning/Testing ● Adversarial testing [Szegedy et al. ICLR’14]: find corner-case inputs imperceptible to human but induce errors 36 School bus OstrichCarefully crafted noise Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Slide adapted from SOSP’17 slides
  • 37. DeepXplore: Automated Whitebox Testing of Deep Learning Systems ‱ Systematic testing of Deep Neural Nets (DNNs) ‱ Neuron coverage: testing coverage metric for deep nerual net ‱ Automated: cross-check multiple DNNs ‱ Realistic: physically realizable transformations (e.g., lighting) ‱ Effective: ‱ 15 State-of-the-art DNNs on 5 large datasets (ImageNet, Self-driving cars, PDF/Android malware) ‱ Numerous corner-case errors ‱ 50% more neuron coverage than existing testing No accident Darker: Accident DeepXplore 37 Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Slide adapted from SOSP’17 slides
  • 38. Example Detected Erroneous Behaviors Turn rightGo straight 38 Go straight Turn left Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE 2018. Slide adapted from SOSP’17 slides
  • 39. Example Detected Erroneous Behaviors Turn rightGo straight 39 Go straight Turn left Pei et al. DeepXplore: Automated Whitebox Testing of Deep Learning Systems. SOSP 2017. Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE 2018. Slide adapted from SOSP’17 slides Lu et al. NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles. CVPR’17. https://arxiv.org/abs/1707.03501“ “
  • 40. Neural Machine Translation Screen snapshot captured on April 5, 2018 ‱ Overall better than statistical machine translation ‱ Worse controllability ‱ Existing translation quality assurance  Need reference translationnot applicable online  Cannot precisely locate problem types and
  • 41. Translation Quality Assurance ● Key ideablack-box algorithms specialized for common problems ○ No need for reference translation; need only the original sentence and generated translation ○ Precise problem localization ● Common problems ○ Under-translation ○ Over-translation Tencent、UIUC Collaborative Work Zheng, Wang, Liu, Zhang, Zeng, Deng, Yang, He, Xie. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv:1807.02340, July 2018.
  • 42. Industry Impact ● Adopted to improve WeChat translation service (over 1 billion users online serving 12 million translation tasks) ○ Offline monitoring (regression testing) ○ Online monitoring (real time selection of best model) ● Large scale test data for translation ○ ~130K English/180K Chinese words/phrases ○ Detect numerous problems in Google Translate and YouDao BLEU Score Improvement %Problems Reduction Problem Cases in Other Translation Services Tencent、UIUC Collaborative Work Zheng, Wang, Liu, Zhang, Zeng, Deng, Yang, He, Xie. Testing Untestable Neural Machine Translation: An Industrial Case. arXiv:1807.02340, July 2018.
  • 43. Quite Many Recent Papers in SE for AI/ML ‱ Ma et al. MODE: Automated Neural Network Model Debugging via State Differential Analysis and Input Selection. ESEC/FSE’18 ‱ Sun et al. Concolic Testing for Deep Neural Networks. ASE’18 ‱ Udeshi et al. Automated Directed Fairness Testing. ASE’18 ‱ Ma et al. DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems. ASE’18 ‱ Zhang et al. DeepRoad: GAN-based Metamorphic Testing and Input Validation Framework for Autonomous Driving Systems. ASE’18 ‱ Dwarakanath et al. Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing. ISSTA’18 ‱ Zhang et al. An Empirical Study on TensorFlow Program Bugs. ISSTA’18 ‱ Tian et al. DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars. ICSE’18 ‱ Abdessalem et al. Testing Vision-Based Control Systems Using Learnable Evolutionary Algorithms. ICSE’18 ‱ Odena, Goodfellow. TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. arXiv:1807.10875. 2018. ‱ 

  • 45. (SE  AI)  Practice Impact Problem Domain Solution Domain Practice Intelligent Software Engineering Intelligence Software Engineering
  • 46. 46 Thank You! Q & A This work was supported in part by NSF under grants no. CNS-1513939, CNS-1564274, CCF-1816615, and a grant from the ZJUI Research Program.