SlideShare a Scribd company logo
Nathan Stoll
nathanstoll@gmail.com
nstoll@google.com
@nathanstoll
Our Product
Web search is great for objective questions, but subjective
questions generate majority of revenues
  Queries     Search quality
  X00B/yr                  RPMs    Examples

                                  •  What is Einstein’s birthday?
    45%        Great       Low
  Objective                       •  How much is the cheapest iPhone?

                                  •  What’s a decent housekeeping service in
    30%                              SF? How much should it cost for a small
  Research                           two bedroom place?

                                  •  What’s a good book to read about
    25%         Ok        High       Romantic Poets?
   Opinion
The Aardvark pitch
Improving a familiar experience
Friends already answer subjective questions, but...
 •  Unreliability of a small number of friends available to you in the moment
 •  Hard to keep up with what your extended network knows about
 •  Social cost of asking for a favor

Why Social Search?
•  Users want personalized responses to questions
•  Most content is still locked in peoples' heads
•  Each individual’s network is growing exponentially
•  Social intimacy makes information actionable
•  Questions about how to spend your time and money are subjective
Our Principles and Process
Minimize risk by maximizing trials
Conception:
•  Picked a problem area
•  Prototyped in serial
•  Abandoned many attempts along the way

Implementation:
 •  User-centered agile engineering
 •  Wizard-of-oz research leading algorithmic automation
 •  Recruited a core team, contracted to fill gaps
 •  Focused on continuous acceleration
Early experimental Aardvark
User-centered agile engineering
Wizard-of-oz research leading algorithmic automation
Team and process learnings
Experiment with process as much as with product
•  Assume you are wrong
•  Make continuous improvement the goal
•  Hire and indoctrinate to support process

Exploit collective wisdom
•  Task people to collect and disseminate learnings
•  Set aside regular time for discussion
•  Make transparency the default
Additional Aardvark reading/viewing
•  White paper on Aardvark Social Search - (
   http://vark.com/aardvarkfinalwww2010.pdf)
•  How we do qualitative user research at Aardvark (
   http://blog.vark.com/?p=314)
•  Our approach to design and development processes (
   http://blog.vark.com/?p=49)
•  Preaching User-Driven Design (
   http://ventilla.posterous.com/preaching-user-driven-design)
•  Startup Lessons Learned Aardvark Case study (
   http://www.justin.tv/startuplessonslearned/b/262666882)
•  Why Machines Need People (
   http://blurkerlab.blogspot.com/2010/04/video-damon-horowitz-at-
   tedxsoma-why.html)

More Related Content

PDF
SCPRSA May 2016 Event: Heather Woolwine - Strenthening Your PR Newsroom
PPTX
Project management for Digital Nomads
PPT
Sinking Or Swimming: The State of Ohio Public Library Web Sites
KEY
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (Th...
PPTX
Webinar: Making the Case for Early STEM Learning- 2016-11-02
PPT
eReader Lending
PPT
Is google wasting your time
PDF
Hypothesis Driven Design: Get Experimental
SCPRSA May 2016 Event: Heather Woolwine - Strenthening Your PR Newsroom
Project management for Digital Nomads
Sinking Or Swimming: The State of Ohio Public Library Web Sites
Fast, Cheap, and Actionable: Creating an Affordable User Research Program (Th...
Webinar: Making the Case for Early STEM Learning- 2016-11-02
eReader Lending
Is google wasting your time
Hypothesis Driven Design: Get Experimental

What's hot (16)

PDF
(Re)mixed Methods for Accessible Product Design
PDF
Designing Data Visualization
PDF
Your Career Unlocked 2014
PPTX
Remote team management - short presentation
KEY
Visualization Requirements Speech
PPTX
Write more effective emails
PDF
Get Better Content with Analytics and User Testing
PDF
Webinar - Coding Clubs: Youth Programming for Public Libraries - 2016-09-28
PPT
User Research on a Shoestring
PDF
Fast, Cheap, and Actionable: Creating an Affordable User Research Program
PDF
Atlassian Q&A - Inside and Out
PDF
Small Project Success
PDF
Small Project? Big Success!
PDF
How To Optimize Your Year End Fundraising
PPTX
I'm a BA Girl in an Agile World @AgileDC 20190923
PPT
Internet librarian v9
(Re)mixed Methods for Accessible Product Design
Designing Data Visualization
Your Career Unlocked 2014
Remote team management - short presentation
Visualization Requirements Speech
Write more effective emails
Get Better Content with Analytics and User Testing
Webinar - Coding Clubs: Youth Programming for Public Libraries - 2016-09-28
User Research on a Shoestring
Fast, Cheap, and Actionable: Creating an Affordable User Research Program
Atlassian Q&A - Inside and Out
Small Project Success
Small Project? Big Success!
How To Optimize Your Year End Fundraising
I'm a BA Girl in an Agile World @AgileDC 20190923
Internet librarian v9
Ad

Viewers also liked (8)

PDF
Scarlet - Scalable, Redundant, Cloud Enabled JIRA
PDF
AtlasCamp 2010: JIRA Plugin Performance Tuning - Alex Hennecke
PDF
K15t - How Atlassian Does Docs
PDF
Administrivia: Golden Tips for Making JIRA Hum
KEY
Kinecting with Wallboards
PPT
JIRA Everywhere
PDF
Enterprise Day 2015 - Comala canvas (Comalatech)
PDF
Revolutionize your IT Team with JIRA Service Desk
Scarlet - Scalable, Redundant, Cloud Enabled JIRA
AtlasCamp 2010: JIRA Plugin Performance Tuning - Alex Hennecke
K15t - How Atlassian Does Docs
Administrivia: Golden Tips for Making JIRA Hum
Kinecting with Wallboards
JIRA Everywhere
Enterprise Day 2015 - Comala canvas (Comalatech)
Revolutionize your IT Team with JIRA Service Desk
Ad

Similar to Users and Algorithms: Incorporating User Centered Design and Advanced Algorithms into Agile Engineering - Atlassian Starter Day 2010 (20)

PPTX
Hands on Usability Testing (Jonathan Rubin)
KEY
Social Search: A Little Help From My Friends
PPT
GAHWNY Spring 2016 Digitization for Historians
PDF
8 Seconds_Writing for Digital Communications.12.11
PPTX
5 minute talks for Reference at the Metcalfe
PPTX
Budget Usability without a Usability Budget
KEY
Publishing Today The Problem
PDF
Webinar: Measuring Program Outcomes: A Toolkit for Small Libraries-2016-05-04
PPTX
a11yTO-Enterprise-Accessibility-Round-Table-Discussion-17NOV2012
PPTX
NYC Open Data Meetup-- Thoughtworks chief data scientist talk
PPTX
Human computation, crowdsourcing and social: An industrial perspective
PDF
Creating a marketing calendar that works for you
PPTX
Career Services 101
PPTX
Campus Connect YourNextLeap - Modern College, Pune (B.Sc.)
PDF
Cultivating Content: Designing Wiki Solutions That Scale
PPTX
Aardvark case study from #sllconf by Max Ventilla and Damon Horowitz
PPTX
SEO Post Panda and Penguin
PPTX
What Questions Are Worth Answering?
PPTX
Running Effective OER and Open Textbook Programs at Your Academic Library: AC...
Hands on Usability Testing (Jonathan Rubin)
Social Search: A Little Help From My Friends
GAHWNY Spring 2016 Digitization for Historians
8 Seconds_Writing for Digital Communications.12.11
5 minute talks for Reference at the Metcalfe
Budget Usability without a Usability Budget
Publishing Today The Problem
Webinar: Measuring Program Outcomes: A Toolkit for Small Libraries-2016-05-04
a11yTO-Enterprise-Accessibility-Round-Table-Discussion-17NOV2012
NYC Open Data Meetup-- Thoughtworks chief data scientist talk
Human computation, crowdsourcing and social: An industrial perspective
Creating a marketing calendar that works for you
Career Services 101
Campus Connect YourNextLeap - Modern College, Pune (B.Sc.)
Cultivating Content: Designing Wiki Solutions That Scale
Aardvark case study from #sllconf by Max Ventilla and Damon Horowitz
SEO Post Panda and Penguin
What Questions Are Worth Answering?
Running Effective OER and Open Textbook Programs at Your Academic Library: AC...

More from Atlassian (20)

PPTX
International Women's Day 2020
PDF
10 emerging trends that will unbreak your workplace in 2020
PDF
Forge App Showcase
PDF
Let's Build an Editor Macro with Forge UI
PDF
Meet the Forge Runtime
PDF
Forge UI: A New Way to Customize the Atlassian User Experience
PDF
Take Action with Forge Triggers
PDF
Observability and Troubleshooting in Forge
PDF
Trusted by Default: The Forge Security & Privacy Model
PDF
Designing Forge UI: A Story of Designing an App UI System
PDF
Forge: Under the Hood
PDF
Access to User Activities - Activity Platform APIs
PDF
Design Your Next App with the Atlassian Vendor Sketch Plugin
PDF
Tear Up Your Roadmap and Get Out of the Building
PDF
Nailing Measurement: a Framework for Measuring Metrics that Matter
PDF
Building Apps With Color Blind Users in Mind
PDF
Creating Inclusive Experiences: Balancing Personality and Accessibility in UX...
PDF
Beyond Diversity: A Guide to Building Balanced Teams
PDF
The Road(map) to Las Vegas - The Story of an Emerging Self-Managed Team
PDF
Building Apps With Enterprise in Mind
International Women's Day 2020
10 emerging trends that will unbreak your workplace in 2020
Forge App Showcase
Let's Build an Editor Macro with Forge UI
Meet the Forge Runtime
Forge UI: A New Way to Customize the Atlassian User Experience
Take Action with Forge Triggers
Observability and Troubleshooting in Forge
Trusted by Default: The Forge Security & Privacy Model
Designing Forge UI: A Story of Designing an App UI System
Forge: Under the Hood
Access to User Activities - Activity Platform APIs
Design Your Next App with the Atlassian Vendor Sketch Plugin
Tear Up Your Roadmap and Get Out of the Building
Nailing Measurement: a Framework for Measuring Metrics that Matter
Building Apps With Color Blind Users in Mind
Creating Inclusive Experiences: Balancing Personality and Accessibility in UX...
Beyond Diversity: A Guide to Building Balanced Teams
The Road(map) to Las Vegas - The Story of an Emerging Self-Managed Team
Building Apps With Enterprise in Mind

Recently uploaded (20)

PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Electronic commerce courselecture one. Pdf
PPTX
A Presentation on Artificial Intelligence
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
NewMind AI Weekly Chronicles - August'25 Week I
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Empathic Computing: Creating Shared Understanding
PDF
Modernizing your data center with Dell and AMD
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PPTX
Big Data Technologies - Introduction.pptx
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
Network Security Unit 5.pdf for BCA BBA.
PDF
Encapsulation theory and applications.pdf
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
Per capita expenditure prediction using model stacking based on satellite ima...
Electronic commerce courselecture one. Pdf
A Presentation on Artificial Intelligence
Advanced methodologies resolving dimensionality complications for autism neur...
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
NewMind AI Weekly Chronicles - August'25 Week I
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
MYSQL Presentation for SQL database connectivity
Empathic Computing: Creating Shared Understanding
Modernizing your data center with Dell and AMD
Mobile App Security Testing_ A Comprehensive Guide.pdf
Big Data Technologies - Introduction.pptx
Reach Out and Touch Someone: Haptics and Empathic Computing
Network Security Unit 5.pdf for BCA BBA.
Encapsulation theory and applications.pdf
Spectral efficient network and resource selection model in 5G networks
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows

Users and Algorithms: Incorporating User Centered Design and Advanced Algorithms into Agile Engineering - Atlassian Starter Day 2010

  • 3. Web search is great for objective questions, but subjective questions generate majority of revenues Queries Search quality X00B/yr RPMs Examples •  What is Einstein’s birthday? 45% Great Low Objective •  How much is the cheapest iPhone? •  What’s a decent housekeeping service in 30% SF? How much should it cost for a small Research two bedroom place? •  What’s a good book to read about 25% Ok High Romantic Poets? Opinion
  • 5. Improving a familiar experience Friends already answer subjective questions, but... •  Unreliability of a small number of friends available to you in the moment •  Hard to keep up with what your extended network knows about •  Social cost of asking for a favor Why Social Search? •  Users want personalized responses to questions •  Most content is still locked in peoples' heads •  Each individual’s network is growing exponentially •  Social intimacy makes information actionable •  Questions about how to spend your time and money are subjective
  • 7. Minimize risk by maximizing trials Conception: •  Picked a problem area •  Prototyped in serial •  Abandoned many attempts along the way Implementation: •  User-centered agile engineering •  Wizard-of-oz research leading algorithmic automation •  Recruited a core team, contracted to fill gaps •  Focused on continuous acceleration
  • 10. Wizard-of-oz research leading algorithmic automation
  • 11. Team and process learnings Experiment with process as much as with product •  Assume you are wrong •  Make continuous improvement the goal •  Hire and indoctrinate to support process Exploit collective wisdom •  Task people to collect and disseminate learnings •  Set aside regular time for discussion •  Make transparency the default
  • 12. Additional Aardvark reading/viewing •  White paper on Aardvark Social Search - ( http://vark.com/aardvarkfinalwww2010.pdf) •  How we do qualitative user research at Aardvark ( http://blog.vark.com/?p=314) •  Our approach to design and development processes ( http://blog.vark.com/?p=49) •  Preaching User-Driven Design ( http://ventilla.posterous.com/preaching-user-driven-design) •  Startup Lessons Learned Aardvark Case study ( http://www.justin.tv/startuplessonslearned/b/262666882) •  Why Machines Need People ( http://blurkerlab.blogspot.com/2010/04/video-damon-horowitz-at- tedxsoma-why.html)