![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Problem Statement
WHAT ISSUE NEEDS SOLVING?
“I am [customer] trying to [accomplish goal].
But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company]
is [adverse business effect]”](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-11-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
I am a 60-year old trying to take a retirement withdrawal online,
because I need to cover the cost of a new car.
But I don’t feel confident doing it online because I don’t
understand if I’m subject to penalties or tax consequences.
This makes me feel anxious.
The consequence for Fidelity is costly customer service calls
and poor customer perception.
How might we…
“I am [customer] trying to [accomplish goal].
But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company] is …](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-12-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise
WHAT ISSUE NEEDS SOLVING?
“I am [customer] trying to [accomplish goal].
But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company]
is [adverse business effect]”](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-13-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
How the issue
will be addressed
What we’ll build
to address it
How we’ll know
it works
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
“[This change] will
enable [who] to achieve
[what], leading to
[value].”
“The simplest way
to test this is to [test
strategy].”
“We will know we’ve
succeeded when we
observe [user
behavior/sentiment].](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-15-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
A Turbo-Tax-like withdrawal process, which guides people
through a decision path, will help customers withdraw retirement
funds with a higher degree of confidence, leading to fewer calls to
the call center.
[This change] will enable [who] to achieve [what], leading to [value].”](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-16-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
The simplest way to test this is with a moderated usability session
+ paper prototype which takes the customer down one sample
decision-making path to completion.
The simplest way to test this is to [test strategy].](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-17-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
We will know we’ve succeeded when we observe customers:
1. Successfully complete the process without indicating they’d
abandon.
2. Indicate that their level of confidence about completing the
transaction is higher than it is today.
We will know we’ve succeeded when we observe [user behavior/sentiment].](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-18-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Hypothesis
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
“[This change] will enable [who] to achieve [what],
leading to [value].”](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-19-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Test Strategy
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
“The simplest way to test this is to [test strategy].”](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-22-320.jpg)
![© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Measurement Strategy
HYPOTHESIS
(solution strategy)
TEST
strategy
MEASUREMENT
strategy
+ +
“We will know we’ve succeeded when we observe [user
behavior/sentiment].](https://image.slidesharecdn.com/labnotebookiasummitshare-160508233109/85/Introducing-The-Lab-Notebook-A-tool-for-managing-a-LEAN-UX-process-31-320.jpg)
Introducing: The Lab Notebook - A tool for managing a LEAN UX process
- 1. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Introducing: The Lab Notebook A hypothesis-driven approach for delivering value faster #UXLabNotebook Julie Casanave @jcasanave Kerry Holeman @uxkerry
- 2. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook What’s this all about?
- 3. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Project completion can easily be mistaken for project success.
- 4. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook A successful launch delivers measurable customer value. If users don’t adopt the solution we need to evaluate and make a decision.
- 5. What was designed & delivered How it’s used
- 6. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Science!
- 7. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook What’s the most important unanswered question? What’s the smallest thing we can make to answer that question? -Steve Mulder, Senior Director of Audience Insights, NPR
- 8. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Create a Lab Report Create the experiment Decide what to do next Define the question
- 9. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Create a Lab Report Create the experiment How will we learn it? Decide what to do next What do we do with what we learned? Define the question What do we want to learn?
- 10. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Question What do we want to learn?
- 11. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Problem Statement WHAT ISSUE NEEDS SOLVING? “I am [customer] trying to [accomplish goal]. But [insert problem] because [insert cause]. This makes me feel [emotion]. The consequence for [my team/company] is [adverse business effect]”
- 12. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook I am a 60-year old trying to take a retirement withdrawal online, because I need to cover the cost of a new car. But I don’t feel confident doing it online because I don’t understand if I’m subject to penalties or tax consequences. This makes me feel anxious. The consequence for Fidelity is costly customer service calls and poor customer perception. How might we… “I am [customer] trying to [accomplish goal]. But [insert problem] because [insert cause]. This makes me feel [emotion]. The consequence for [my team/company] is …
- 13. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Exercise WHAT ISSUE NEEDS SOLVING? “I am [customer] trying to [accomplish goal]. But [insert problem] because [insert cause]. This makes me feel [emotion]. The consequence for [my team/company] is [adverse business effect]”
- 14. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Experiment How will we learn?
- 15. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Experiment How the issue will be addressed What we’ll build to address it How we’ll know it works HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + “[This change] will enable [who] to achieve [what], leading to [value].” “The simplest way to test this is to [test strategy].” “We will know we’ve succeeded when we observe [user behavior/sentiment].
- 16. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Experiment HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + A Turbo-Tax-like withdrawal process, which guides people through a decision path, will help customers withdraw retirement funds with a higher degree of confidence, leading to fewer calls to the call center. [This change] will enable [who] to achieve [what], leading to [value].”
- 17. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Experiment HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + The simplest way to test this is with a moderated usability session + paper prototype which takes the customer down one sample decision-making path to completion. The simplest way to test this is to [test strategy].
- 18. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Define the Experiment HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + We will know we’ve succeeded when we observe customers: 1. Successfully complete the process without indicating they’d abandon. 2. Indicate that their level of confidence about completing the transaction is higher than it is today. We will know we’ve succeeded when we observe [user behavior/sentiment].
- 19. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Exercise: Hypothesis HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + “[This change] will enable [who] to achieve [what], leading to [value].”
- 20. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide on a Test Strategy SENTIMENT (Whatpeoplesay) BEHAVIOR (Whatpeopledo) QUANTITATIVE (Reveals the “what”) QUALITATIVE (Reveals the “why”) Surveys Moderated Usability Study A/B Testing Self/Un-Moderated Usability Study Opinion Lab Verbatims Focus Groups Interview Call Center Notes Social Media Activity Web Analytics Emotional Reaction Study Card Sort Treejack Study First Click Study Eye-Tracking Study Pre-release Post-release PRO TIP Be biased towards behavioral methods
- 21. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Think creatively about what type of experiment will give the results you need. Deployed code is only one kind of experiment. (and often the most expensive kind)
- 22. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Exercise: Test Strategy HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + “The simplest way to test this is to [test strategy].”
- 23. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden Measures of attitudes, often collected via survey • Emotional reaction • Perceived level of effort • Sentiment rating • Level of confidence • Volume of opinion lab comments
- 24. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden Level of user involvement • Bounce rate • Channel switching • Page visits • Searches • Click-throughs • Eye-tracking • Visitors
- 25. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden Gaining new users of a product or feature • Click rates • Click-throughs • New accounts
- 26. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden The rate at which existing users are returning • NA for our program
- 27. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden Efficiency, effectiveness, and error rate • Completion speed • Error rate • Abandonment • Completion rate • Pathing • Re-searches
- 28. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden • Emotional reaction • Perceived level of effort • Sentiment rating • Bounce rate • Channel switching • Click rates • Click-throughs • Not applicable • Completion speed • Error rate • Level of confidence • Volume of opinion lab comments • Page visits • Searches • Click-throughs • Abandonment • Completion rate • Eye-tracking • Visitors • New accounts • Pathing • Re-searches What metric matters most?
- 29. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden • Emotional reaction • Perceived level of effort • Sentiment rating • Bounce rate • Channel switching • Click rates • Click-throughs • Not applicable • Completion speed • Error rate • Level of confidence • Volume of opinion lab comments • Page visits • Searches • Click-throughs • Abandonment • Completion rate • Eye-tracking • Visitors • New accounts • Pathing • Re-searches Does this experience give you a higher or lower level of confidence compared to today’s process?
- 30. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide How to Measure HAPPINESS ENGAGEMENT ADOPTION RETENTION TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden • ?? • ?? • ?? • ?? • ?? • ?? • ?? • ?? • ?? • ?? • ?? WHICH ONE MATTERS MOST? WHAT’S YOUR SET OF METRICS?
- 31. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Exercise: Measurement Strategy HYPOTHESIS (solution strategy) TEST strategy MEASUREMENT strategy + + “We will know we’ve succeeded when we observe [user behavior/sentiment].
- 32. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Decide what to do next What do we do with what we learned?
- 33. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook What We Learned Major findings, unanticipated discoveries Pivot? Or Persevere? Or Stop? At what point was the conclusion reached? Time Lapsed Since Launch Conclusion What will happen next?Next steps DD Month YYYYLaunch Date Documenting what was learned Outcomes How did experiment do against HEART metrics?
- 34. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook A disproven hypothesis is evidence. It is not failure. “I have not failed. I've just found 10,000 ways that won't work.” -Thomas A. Edison
- 35. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook And when that experiment is done?
- 36. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook LATHER, RINSE, REPEAT ? ! ? ! ? ! ? ! ? ! ? ! DESIGN RUN ITERATE LABNOTEBOOK LAB REPORT LAB REPORT LAB REPORT LAB REPORT
- 37. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook The “grain of salt” slide
- 38. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook MITIGATE RISK • Deliver value faster • Reduce wasted time, effort, money FOCUSED EXPERIMENTS • Clear learning agenda • Simple test strategy • Well defined metrics
- 39. © 2016 Proprietary material; do not use without permission of author | #UXLabNotebook Thank You. Julie Casanave | @jcasanave Kerry Holeman | @UXKerry
Editor's Notes
- #29: This is a framework that lets us identify how we’re going to determine success. While the business will continue to have high-level KPIs and performance scorecards, this framework takes it down to a more granular level, and gives us some very concrete ideas about how to determine if we’ve delivered actual value. It lets us look both at sentiment and behavior, both of which tell us about how much people like using our stuff.
- #30: This is a framework that lets us identify how we’re going to determine success. While the business will continue to have high-level KPIs and performance scorecards, this framework takes it down to a more granular level, and gives us some very concrete ideas about how to determine if we’ve delivered actual value. It lets us look both at sentiment and behavior, both of which tell us about how much people like using our stuff.