Data-Driven Software Engineering for Agile Teams

AW2
Concurrent Session
11/11/15 11:30am

“Data-Driven Software Engineering for Agile
Teams”

Presented by:
Viktor Veis
Microsoft

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

Viktor Veis
Microsoft
A software engineering manager with Visual Studio, Viktor Veis focuses on telemetry and data-driven
engineering. Viktor built the Visual Studio telemetry team from the ground up and drives transformation to
agile data-driven culture across Visual Studio feature engineering teams. He has led performance strategy
in Bing where a few milliseconds in page load time results in big changes in revenue. Viktor has
experience in v1 projects such as Photosynth, establishing a performance culture from the beginning.
Viktor was previously a developer in several small companies on both sides of the Atlantic, building entire
application stacks including firmware, communication protocols, APIs, and user experience. Read
Viktor’s blog.

1
Data-Driven Software Engineering
for Agile Teams
Viktor Veis
Microsoft, Visual Studio
Data-Driven Software Engineering for Agile Teams
Decisions

2
Plan for the Next Hour
Telemetry data + Math you already know
Better and faster decisions
Backlog prioritization
Performance and reliability optimizations
Reduce test cost
Measure true impact of a feature
Input
Output
Feature Impact

3
Intuition Driven Decisions
Intuition Driven DecisionsIntuition Driven Decisions

4
Intuition Driven DecisionsIntuition Driven Decisions
HiPPO

5
Bing Experiment
Color Scheme A Color Scheme B
http://www.exp-platform.com/Pages/SevenRulesofThumbforWebSiteExperimenters.aspx
Intuition Driven DecisionsBing Experiment
Color Scheme A Color Scheme B
http://www.exp-platform.com/Pages/SevenRulesofThumbforWebSiteExperimenters.aspx

6
More Bing Experiments
http://aka.ms/BingExperiments
Usability Studies and Customer Feedback
Bias
Small sample size
Most vocal part of the community
Measurement effect
CC BY-ND 2.0 by John Crel

7
Usability Studies and Customer Feedback
Viktor’s Journey
ServiceStartup Application

8
Takeaway #1
Startups, services and apps
can make better and faster decisions
with data
Takeaway #2
Developers are good in dealing with one
data point at a time

9
Takeaway #2
Developers are good in dealing with one
data point at a time
Data analysis at scale is a new skill
Evolution of Feature Complete
Historical
Works Locally
Checked In

10
Historical
Works Locally
Checked In
E2E Test Automation
Live Site
Combined Engineering
Historical
Works Locally
Checked In
E2E Test Automation
Live Site
Data Driven Engineering
Used by Thousands* Daily
Fast and Reliable

11
Historical
Works Locally
Checked In
E2E Test Automation
Live Site
Data Driven Engineering
Used by Thousands* Daily
Fast and Reliable
Experiment Driven Engineering
Proven to Increase
Business Metric Y by X
Build Measure Learn
https://www.safaribooksonline.com/library/view/running-lean-2nd/9781449321529/ch01.html

12
Build Measure Learn
Our Focus
Usage Trend

13
Usage Trend
Telemetry Infrastructure
Instrumentation

14
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
Instrumentation
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
Telemetry Service
Instrumentation

15
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
StorageTelemetry Service
Instrumentation
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
Instrumentation Query

16
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
Instrumentation and Query
is the only code you need to write

17
Telemetry API
• Timestamp
• User ID
• Persistence
• Uploader
Working With Data
https://en.wikipedia.org/wiki/Multitier_architecture

18
Telemetry Data != Product Data
http://datadrivenengineering.com/2015/04/22/revisiting-software-architecture-principles-for-telemetry-data/
Custom Dashboard – Really?
https://www.bing.com/images/search?q=dashboard

19
Instrumentation and Metrics
Instrumentation
An API call to report that something
interesting happened
Metric
A result of querying the data reported by
instrumentation
Instrumentation and Metrics
•User Actions
•Latency
•Errors
•User Count
•Click Count
•Latency Percentile
•Error Count and Rate
•Time to Complete
•Success Rate
Instrumentation Metrics

20
Metrics for Business
•Acquisition
•Engagement
•Churn
•Revenue
•…
Build Measure Learn
Who?

21
Build Measure Learn
DeveloperData Scientist
I do math
I write
code
App Example

22
Instrumented App
Telemetry
Data
StorageInstrument
Telemetry Data

23
Data Tools
Data Tools

24
Build Measure Learn
Complex
DeveloperData Scientist
Build Measure Learn
Developers shouldn't have to rely on
data scientists to get value out of data

25
Telemetry Data - Timestamp
Filter: 7 days
Telemetry Data - Action
Group By
Action Count
Purchase 2500
Subscribe 700
Share 20
Last 7 Days

26
Telemetry Data - Users
Distinct Count
Action Count Users Per User
Purchase 2500 1000 2.5
Subscribe 700 100 7
Share 20 20 1
1050 Unique UsersLast 7 Days
Telemetry Data - Latency
Action Count Users Per User Latency-P75 Latency-P95
Purchase 2500 1000 2.5 1500 2500
Subscribe 700 100 7 1200 7000
Share 20 20 1 5000 9000
1050 Unique UsersLast 7 Days
Aggregate

27
Insights
1. Purchases are healthy
2. Users do not share
3. Sharing is slow
4. Too many subscriptions from the same user
5. Subscribe latency has a long tail
Purchase 2500 1000 2.5 1500 2500
Subscribe 700 100 7 1200 7000
Share 20 20 1 5000 9000
1
2 3
4 5
Daily Trend
Purchase 2500 1000 2.5 1500 2500

28
Data Analysis 101
Select a date range divisible by 7
Group by user actions
Number of unique users
Count user actions per user
Percentiles for latencies
Daily trends with 7-day moving average
Debugging Data
Why??

29
What Changed?
What Changed?Brainstorm, Research & Validate
Segments
What else?
Version
Browser
Country
…

30
Segments
Search for segments that are large and
different. Target them to deliver unique
value.
CC BY-SA 3.0 by Reinhard KraaschCC BY 2.5 by Jo Weber
Segments

31
Histogram
Purchase 2500 1000 2.5 1500 2500
Histogram
Purchase 2500 1000 2.5 1500 2500

32
Histogram
Purchase 2500 1000 2.5 1500 2500
Histogram
Action Count Per User Latency-P75 Latency-P95
Purchase 2500 2.5 1500 2500
Second Peak
Long Tail

33
Debugging Data
What changed?
Segments that are large and different
Histograms for abnormal data shape
Sub-metrics
Data Tests
ROI
Test Lab Data Tests

34
Data Tests
Proven to Increase
Business Metric Y by X

35
Correlation vs Causation
https://commons.wikimedia.org/wiki/File:PiratesVsTemp(en).svg

36
Price Reduction
Marketing Competition
New Products
Going Viral
A/B Experiments
All Users
50% 50%
Metric White Pink P-Value
Average Purchase Count per User Per Week 2.2 2.5 0.0008
…

37
P-Value
https://en.wikipedia.org/wiki/P-value
P-Value
•Chance of getting results we did (or more extreme)
assuming no change in user behavior between A and B
•Between 0 and 1
•Lower p-value increases confidence that the change in
behavior is real and sustainable
•<0.05 – Typical threshold for believing in change

38
P-Value in Excel
=T.TEST(A:A,B:B, 2, 2)
Summary
Make data analysis a part of your daily agile process
Formal stats training is optional to get started
Reuse infrastructure to start fast
Data tests can replace expensive lab tests
A/B experiments measure true impact of a feature

39
Where is Your Team?
Data-driven engineering:
• How many users does this feature have?
• What are the main entry points and segments?
• Do we measure success based on telemetry data?
• Do we learn about issues from data and not from upset
customers?
Experiment-driven engineering:
• How do we measure feature impact?
• How do we make release/cut decisions?
• How do we prioritize performance against feature work?
Thank You
Blog: http://www.DataDrivenEngineering.com
Twitter: @ViktorVeis

Data-Driven Software Engineering for Agile Teams

More Related Content

What's hot (20)

Similar to Data-Driven Software Engineering for Agile Teams (20)

More from TechWell (20)

Recently uploaded (20)

Data-Driven Software Engineering for Agile Teams