Lean Analytics - Communication Layer into the World of Business

Lean Analytics
The Communication Layer into the
World of Business
Budapest, 2016
@byosko

Agenda
09:00-11:00 - Tools & frameworks for “talking business”
• Lean Startup (it’s not just for startups!)
• Business Mapping (Lean Canvas)
• Product Management & Data (roadmaps & feature prioritization)
11:00-11:30 - Coffee break
11:30-12:30 - Lean Analytics: The basics
• Making data simpler for everyone else
• What makes a good metric
• Types of metrics
12:30-13:30 - Lunch
13:30-15:00 - Lean Analytics II
• Lean Analytics framework
• One Metric that Matters
• Lean Analytics Cycle
15:00-15:30 - Coffee break
15:30-17:30 - Interactive session & Q&A

TOOLS &
FRAMEWORKS FOR
“TALKING BUSINESS”1

DATA IS A
COMMUNICATION TOOL
http://www.instigatorblog.com/data-common-language/2016/09/22/

@byosko
WE HAVE TO TALK
BUSINESS FIRST

TALKING BUSINESS: LEAN STARTUP

LEAN STARTUP ISN’T JUST
FOR STARTUPS

ENTERPRISES ARE GOING LEAN…
“By 2021, more than 50% of established corporations will
be leveraging lean startup techniques.” (Gartner)

Core Adjacent Transformative
Do the same thing
better.
Nearby product, market,
or method.
Start something
entirely new.
Regional 
optimizations.
Innovation, go-to-market
strategies.
Reinvent the business
model.
• Get there faster
• Smaller batches
• Solution, then testing
• Increased accountability
• Customer development
• Test similar cases
• Parallel deployment
• Analytics & cycle time
• Fail fast
• Skunkworks/R&D
• Focus on the search
• Ignore the current model &
margins
Many models for enterprise innovation

Core Adjacent Transformative
Know the problem
(customers tell you it)
Know the solution
(customers/regulations/
norms dictate it.)
Know the problem (market
analysis)
Don’t know the solution (non-
obvious innovation confers
competitive advantage.)
Don’t know the problem (just
an emerging need/change)
Don’t know the solution.
Waterfall: 
Execution
matters
Agile/scrum: 
Iteration
matters
Lean Startup:
Discovery
matters
Another way to look at it

Current 
state
Business
optimization
Product, 
market, 
method
innovation 
Business
model
innovation 
You can convince
executives of this
because some of it
is familiar.
This terriﬁes them
because it eats the
current business.
A three-maxima model for enterprise
innovation

Improvement Adjacency Remodelling
Do the same, 
only better.
Explore what’s 
nearby quickly
Try out new 
business models
Lean approaches apply, but the metrics vary widely.
Sustain /
core
Innovate /
adjacent
Disrupt /
transformative

Sustaining Adjacent Disruptive
Next year’s car Electric car, 
same dealer
On-demand, app-based 
car service

LET’S GO BACK TO HOW
LEAN WORKS…

Don’t sell what you can make. Make what you can sell.
Kevin Costner is a lousy entrepreneur.

The core of Lean
is iteration.

Key principles of Lean Startup
• “Get out of the building”
• Customer discovery is key
• Identify and solve the riskiest thing ﬁrst
• Reduce waste
• Most important: learning

“THERE ARE NO
ANSWERS INSIDE
THE BUILDING.”
- Steve Blank, author of Four Steps to the Epiphany

Build Measure Learn
• What do I build?
• How do I know if it’s the
right thing?
• How do I start?

START WITH A PROBLEM:
What problem are you trying
to solve?

Fall in love with
the…
PROBLEM
…not the solution.

LEAN STARTUP IS ABOUT
FINDING THE TRUTH,
AND DOING SO AS
QUICKLY AS POSSIBLE.

SO HOW DO WE FIGURE
OUT WHAT PROBLEMS
TO SOLVE?

http://talkingtohumans.com https://leanstack.com/running-lean-book/ http://keytakeaways.io/books/four-steps-epiphany/
A few good references on customer development, problem/
solution interviews and more

TALKING BUSINESS: ASSUMPTIONS
Understand your business assumptions
• Make a list of your riskiest
assumptions ﬁrst
• Assumptions have to be
testable
• “We believe that…”
Do you know the deﬁnition of
“Assume”?

BUSINESS ASSUMPTIONS Day 2 - Lean Startup
ZxLERATOR | NYC | SUMMER 2016 31
• My target customer will be? (Tip: how would you describe your primary
target customer)
• The problem my customer wants to solve is? (Tip: what does your
customer struggle with or what need do they want to fulﬁll?)
• My customer’s need can be solved with? (Tip: give a very concise
description / elevator pitch of your product)
• Why can’t my customer solve this today? (Tip: what are the obstacles
that have prevented my customer from solving this already?)
• The measurable outcome my customer wants to achieve is? (Tip:
what measurable change in your customer’s life makes them love your
product?)
• My primary customer acquisition tactic will be? (Tip: what is your
current guest for the top 1 or 2 marketing channels?)

BUSINESS ASSUMPTIONS Day 2 - Lean Startup
• My earliest adopter will be? (Tip: remember that you can’t get to the
mainstream customer without getting early adopters ﬁrst)
• I will make money (revenue) by? (Tip: don’t list all the ideas for making
money, but pick your primary one)
• My primary competition will be? (Tip: think about both direct and
indirect competition?)
• I will beat my competitors primarily because of? (Tip: what truly
differentiates you from the competition?)
• My biggest risk to ﬁnancial viability is? (Tip: what could prevent you
from getting to breakeven? Is there something baked into your revenue
or cost model that you can de-risk?)
• My biggest technical or engineering risk is? (Tip: is there a major
technical challenge that might hinder building your product?)

WHAT ASSUMPTIONS DO
YOU HAVE, IF PROVEN
WRONG, WOULD CAUSE
YOUR BUSINESS TO FAIL?

TALKING BUSINESS: MAPPING YOUR BUSINESS
• It’s difﬁcult to create value for a business without understanding how
the business functions
• There are several systems for designing/mapping a business
• The goal is to understand all the actors involved, their motivations,
interests and biases
• Common questions you’ll want to answer: Where does the money
come from? Who is the customer? Who are the partners?
The importance of mapping your business

Does recurring
revenue work
for everyone?
CASE STUDY

The leader in predictive analytics for people. Clearfit
helps thousands of companies build better teams. As
featured in:
CASE STUDY
10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers or
hiring. You have to understand your customer, who they are,
how and why they buy, and how they value your product or
service.” - Ben Baldwin

Business Model Canvas:
http://www.businessmodelgeneration.com/canvas/
bmc
Lean Canvas:
https://leanstack.com/lean-canvas/
Two good approaches

http://leanstack.com/lean-canvas/
• A 1-page “Business Plan”
• Helps identify key aspects of your business and biggest risks
• Should be doable in 20-minutes
• A visualization of the business assumptions list from Talking
to Humans
• Ideally you update it over time at major milestones / key
discoveries
Introducing Lean Canvas

Lean Canvas
Day 3 - Business Models &
Value Propositions
DESIGNING A BUSINESS MODEL
1
2
3
4
5
67
8
9

http://leanstack.com/lean-canvas/
Lean Canvas Best Practices
1
2
3
Try and test one thing at a time
Have a hypothesis around how to test each section
and decide on an MVP to experiment with
Update the Lean Canvas whenever you’ve learned
something signiﬁcant

1. Testing the Problem
• Ongoing customer discovery; problem interviews -- largely
collecting qualitative feedback
• Remember: No one pays for their 5th problem to be solved
• For existing alternatives deﬁne one clear, direct competitor.
• Consider other ways users/customers can address their problems
• What products or services exist as alternatives to what you’re offering?
• Be careful about problems that are too high level (“universal
truths”)

2. Identifying Customer Segments
• Define 3-4 specific user personas for your early adopter groups
• Who is your intended audience?
• What type of person do you anticipate benefiting most from your
product?
• Don’t forget there may be a difference between the users of your product
and the buyer
• For early adopters: what makes them different?
• Understand your best users: http://www.instigatorblog.com/your-best-users/
2014/06/20/

3. Testing the Unique Value Proposition
• What do you do, why are you different, and why are you worth
investing in?
• What’s the high level concept?
• Conduct solution interviews (this is where you’ll start to test your
UVP)
• Create landing pages with different messaging

More on Value Propositions
Steve Blank’s XYZ
Template: “We help X do Y doing Z”.
Sample: We help non-technical
marketers discover return on investment
in social media by turning engagement
metrics into revenue metrics
Dave McClure’s Elevator Ride
Template:
• Short, simple, memorable: what, how, why.
• 3 keywords or phrases
• KISS (no expert jargon)
Sample: Mint.com is the free, easy way to
manage your money online.
http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-
propositions-that-work/

4. Testing Solutions
• Conduct solution interviews — get qualitative feedback from
users/customers
• Measure interest in solutions (before building anything)
• Start mapping out your MVPs
• Map key features to key problems

5. Testing Channels
• How will users/customers come in contact with your brand?
Where will they first learn about your business?
• Think about the various touch points before people buy, during
purchase, and after
• Don’t worry too much about cost (initially), focus more on
channel success in terms of conversion and quality of users/
customers

6. Testing Revenue
• How will you make money? Who will pay?
• Early on, revenue is a proxy of value creation, but now you have
to test the economic engine of the business more seriously
• Remember: users of your product may not be the customers, or
you may have multiple customers (e.g. top-down enterprise
software)
• Metrics of importance: ARPU, LTV, CAC, MRR, sales, etc.

6. Testing Pricing
http://download.red-gate.com/ebooks/DJRTD_eBook.pdf
“Prices are a shortcut to our most sensitive emotional
responses.” – Tom Whitwell
https://medium.com/dreamit-perspectives/founders-guide-to-product-pricing-
b187093e2483#.vdpwcl752
! Don’t overcomplicate too early
! Sell based on aspirational value (not negative value)
! Don’t ask customers what they’d pay, they don’t know
! Test it (over and over again)

7. Measuring Costs
• What are the costs?
• Where do they come from?
• How are they impacted by growth/scale?
• For Web-related businesses, costs are usually low and not
worth focusing on initially; but for other types of businesses
(manufacturing, hardware) costs are much higher. For Web-
related businesses, costs are mostly in acquisition (where LTV >
CAC becomes important)

8. Metrics
• We’ll discuss this in more detail later today!

9. Deﬁning an Unfair Advantage
• What makes you stand out from competitors?
• What do you know or have access to that no one else does?
• This is very hard to come up with; there are very few legitimate
unfair advantages
• Insider information
• Single-minded obsession with the “One Thing”
• Personal authority
• Existing customers / distribution
http://blog.asmartbear.com/unfair-advantages.html

Value Propositions
LEAN CANVAS
Example Lean Canvas: Freckle
http://blog.asmartbear.com/unfair-advantages.html
http://www.slideshare.net/de-pe/lean-canvas-process-and-examples

1
2
3
Fill out your Lean Canvas in 20 minutes
Think through the business assumption
questions from Talking to Humans
We’ll share and discuss after
Time to give it a try!

Product management is
the “glue” between
everyone and everything
TALKING BUSINESS: PRODUCT MANAGEMENT

PRODUCT MANAGEMENT IS THE
PROCESS BY WHICH WE TURN
VISION INTO REALITY
PRODUCT MANAGEMENT =
STRATEGY - PRIORITIZATION EXECUTION

The role of product managers
• Product managers are the “glue” between all stakeholders
(internal & external)
• Work towards achieving the company’s vision through strategy
& execution
• Empower others to get things done
• Prioritize feature development based on all the inputs
• Ensure deliverability as expected (on time)
• Measure results (success or failure)

“The job of a product manager is
to: Help your team (and company)
ship the right product to your
users.”
- Josh Elman
https://medium.com/@joshelman/a-product-managers-job-63c09a43d0ec#.5mhlul6l1

DATA IS A KEY INPUT AND
FILTER IN PRODUCT
MANAGEMENT

COMPETITION,
OTHER PRODUCTS,
BEST PRACTICES
BUILDLEARN
IDEAS
CORPORATE
GOALS (SOME GOOD,
SOME BAD)
GUTS & INSTINCTS
PARTNERS
OTHER
DEPARTMENTS
INDUSTRY
TRENDS, ETC.
DATA
CUSTOMER INPUT
DATA

COMPETITION,
OTHER PRODUCTS,
BEST PRACTICES
PARTNERS
INDUSTRY
TRENDS, ETC.
GUTS & INSTINCTS
OTHER
DEPARTMENTS
CORPORATE GOALS
DATA
AS A
FILTER
BETTER
DECISIONS
CUSTOMER INPUT

@byosko
Company vision
Internal and
external inputs
Perpetual problem / solution validation
Project Scope Creation
Sprints
Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog
How it all comes together

@byosko
Company vision
Internal and
external inputs
Perpetual problem / solution validation
Project Scope Creation
Sprints
Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog
How it all comes together
DATA HAS AN IMPACT ON EVERY STEP IN THE
PRODUCT DEVELOPMENT PROCESS

Product roadmaps (problem discovery)
Product roadmaps need to answer
three critical questions:
1. Where are we going? (Vision)
2. Why are we going there? (Business
Goals / Value Creation)
3. And how are we going to do it?
(Resources & Planning)

Best practices for great product roadmaps
DON’T
FORGET
TO KISS!
• Focus on goals/outcomes not features
• Categorize and organize by themes
• Make sure problems are clearly defined and
understood
• Timelines are loose at best (this isn’t the core focus)
• Use as a conversation starter, not as something writ
in stone

Identifying the core problems / goals (objectives)
• Collect and analyze feedback / inputs
• Identify the core problems (goals) for the business
(problem discovery)
• Identify the One Metric That Matters for each
problem/goal and draw a line in the sand
• Hypothesize potential solutions
• Be outcome driven

Collecting input
Product & Design
(defining goals /
objectives)
User & customer
feedback
Sales
Marketing
Customer Support
Etc.
! In-person interviews
! Surveys
! Customer support
inquiries
! Real-time online
Supported
by data
Your gut
Company vision
Your own ideas

Collecting input
• Do you have a customer panel? (e.g. private FB
group, mailing list, etc.)
• Can you schedule regular in-person sessions?
• Be wary of salespeople suggesting, “If we only built
feature X, I could close this deal.”
• Attend customer and sales meetings; spend time on
the help desk.
• Work with the product team to segment your user/
customer base.

Intercom’s product roadmap inputs
“Building a great product
is an art as much as a
science. It requires
making hard decisions
and trade-offs, in
circumstances ranging
from being overwhelmed
with data to having no
data.”
https://blog.intercom.io/where-do-product-roadmaps-come-from/

Roman Pichler’s GO Product Roadmap
! Blends high-level roadmap and
release planning
! Name can function as a theme
! Goal is a high-level purpose
aligned with the stage you’re at
(e.g. Stickiness, Virality, etc. or
use AARRR)
! Roadmap leads to epics,
stories, specs & MVPs
http://www.romanpichler.com/blog/working-go-
product-roadmap/
The release date or
timeframe
The reason for
creating the release
The metrics to
determine if the goal
has been met
The high-level
features necessary to
meet the goal
The name of the new
release
Date or timeframe Date or timeframe Date or timeframe Date or timeframe
Name / version Name / version Name / version Name / version
FeaturesFeaturesFeaturesFeatures
Goal Goal Goal Goal
MetricsMetricsMetricsMetrics

Feature prioritization
1. Start with the key goals / outcomes & metrics
2. Rank order the solutions (features) based on value creation
a. Identify a “line in the sand” for each solution (target)
3. Rank order the solutions based on effort
Ultimately, what will create
the MOST value for the LEAST effort?

Idea / Solution Score Target Engagement
Value
Effort
Send more push notifications at regular
intervals regarding key actions we want users
to take inside the app
9 +10% MAU 3 1
Improve first user experience by providing
users with a key action they need to take right
away (no “white screen of death”)
5 +15% MAU 5 3
Add more robust reporting features for date
range searches, saved reports
1.5 +5% MAU 2 4
Leaderboard for users hitting sales targets 1.5 +3% MAU 1 2
Goal: Increase Engagement
(target: +15% MAU)
Try using a “dot voting”
system per idea/solution
Only a rough estimate is
needed here; goal is to
have comparative values
@byosko@byosko
Use a simple weighted
formula to score/rank items
(e.g. Engagement Value *
3 / Effort)
Sample ranking system

TALKING BUSINESS: PRODUCT MANAGEMENT https://blog.intercom.io/rice-simple-prioritization-for-product-managers/
! Reach: How many people will be impacted by the new
feature over a given period of time?
! Impact: How much will this new feature impact an
individual user/customer? 3 for “massive impact”, 2 for
“high”, 1 for “medium”, 0.5 for “low” and finally 0.25 for
“minimal”.
! Confidence: How confident are we in our estimates?
100% is “high confidence”, 80% is “medium”, 50% is
“low”.
! Effort: How much time will it take to deliver the new
feature? Measure in “person months”.
Reach x Impact x Confidence
Effort
RICE
Score
Intercom’s RICE model for feature prioritization

Data’s role in product management
• As an input for discovering interesting trends that are
worth pursuing in product development
• As a filter for validating the relative importance of
features
• As a check throughout the entire product
development process from establishing a product
roadmap, to feature prioritization, to scope creation
and sprint management
• This is all about effective communication

MAKING DATA
SIMPLER FOR
EVERYONE ELSE2

NOW THAT WE KNOW HOW
TO “TALK BUSINESS” IT’S
TIME TO “TALK DATA”

YOU AND YOUR COWORKERS
SPEAKING DATA
https://www.ﬂickr.com/photos/jdhancock/8031897271

LEAN ANALYTICS: THE BASICS
1
2
3
Think about a project or product you’re involved
with today
Can you write down the key metrics that
matter?
We’ll compare after going through the next
section
Quick exercise

Analytics is the measurement of
movement towards business goals.

A GOOD METRIC IS:
Understandable
If you’re busy explaining
the data, you won’t be
busy acting on it.
Comparative
Active Users vs.
Active Users/month
A ratio or rate
% Monthly Active
Users
Behavior changing
You know how you’ll
change your business
based on what the
metric tells you.

LEAN ANALYTICS: THE BASICS Day 3 - Lean Analytics
Herbert Simon
If a metric won’t
change how you
behave, it’s a…
BAD
METRIC

Metrics help you know yourself
Acquisition
Hybrid
70% 
of retailers
You are just
like
Customers that buy
>1x in 90d
Once
2-2.5 
per year
Your customers will
buy from you
Then you are in
this mode
1-15%
15-30%
Low acquisition cost,
high checkout
Focus on
20% 
of retailers
Increasing return
rates, market share
Loyalty>30%
Loyalty, selection,
inventory size
>2.5 
per year
10% 
of retailers
(Thanks to Kevin Hillstrom for this.)

VANITY vs. ACTIONABLE
Vanity Actionable
Makes you feel good but
doesn’t change how you’ll act.
Helps you pick a direction and
change your behavior.
“Up and to the right” These are good.

QUALITATIVE vs. QUANTITATIVE
Qualitative Quantitative
Unstructured, anecdotal,
revealing, hard to aggregate.
Numbers and stats; hard
facts, but less insights.
Warm and fuzzy. Cold and hard.

DISCOVER QUALITATIVELY.
PROVE QUANTITATIVELY.

Do Airbnb hosts get more
business if their property
is professionally
photographed?

Case study: Does professional photography
make a difference?
Gut instinct (hypothesis)
Professional photography helps Airbnb’s business
Built a Concierge MVP
Sent 20 photographers out into the ﬁeld
Measured results
Compared photographed listings to control group
Made a decision
Launched photography as a new feature to all hosts

EXPLORATORY vs. REPORTING
Exploratory Reporting
Speculative. Tries to ﬁnd
unexpected insights. Source
of unfair advantage.
Predictable. Keeps you
abreast of normal, day-to-day
operations. Can be managed
by exception.
Cool. Necessary.

Case study: Finding insights in the data
• Started as Circle of Friends
• Leveraged Facebook early
• Grew to 10M users fast
ENGAGEMENT SUCKED!

Case study: Moms are crazy (but in a good
way!)
ENGAGEMENT SOLVED!
• Messages to one another were ~50% longer
• 115% more likely to attach a picture to a post
• 110% more likely to engage in a threaded conversation
• Invited friends were 50% more likely to become engaged users
• 60% more likely to accept invitations to the app

LAGGING vs. LEADING
Lagging Leading
Historical metric that shows
you how you’re doing: reports
the news.
Number today that shows a
metric tomorrow: makes the
news.
Start here. Try and get here.

Examples of leading metrics
A Facebook user reaching 7 friends within 10 days of signing up.
(Chamath Palihapitiya)
A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi
Wang)
A Twitter user who follows a certain number of people, and a certain
percentage of those people follow the user back. (Josh Elman)
A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)

Case study: Buffer’s leading metrics
revealed
Buffer discovered 3 leading metrics for long-term engagement:
1 People who install the Chrome extension
2 People who connect more than 1 social account
3 People who share 15 pieces of content in 7 days

Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlation vs. causation

CORRELATED vs. CAUSAL
Correlated Causal
Two variables that are related
(but may be dependent on
something else.)
An independent variable that
directly impacts a dependent
one.
Ice cream and
drowning.
Ice cream and summertime.
Drowning and summertime.

Cohort analysis

Case study: Putting basic data to use
Ricky (product manager) had some ideas for improving the “Proposal Send Screen” (based on
qualitative feedback & gut), but before prioritizing this work, he digs into the data.
• 50% of people send proposals through Proposify
(50% don’t) (quantitative) — Is this good or bad?
Ricky isn’t sure. So he’s going to need to look at
additional data (exploratory):
• Churn
• Size of customer
• Proposal won rate
• Any correlations here?
• Can also do some direct customer developer to
learn more (qualitative)
• Might lead to additional, meaningful product dev
(actionable)

1
2
Look at the metrics you wrote down before,
how many of them stand up?
How can you make your metrics simpler for
people to understand?
Quick exercise

LEAN ANALYTICS:
THE FRAMEWORK
3

LEAN ANALYTICS: THE FRAMEWORK
Lean Analytics Framework
What business
are you in?
What stage are
you at?
• E-Commerce
• SaaS (freemium)
• Mobile app (gaming)
• Two-sided marketplace
• Media
• User-generated content
• Empathy
• Stickiness
• Virality
• Revenue
• Scale

Dave McClure’s Pirate Metrics

Eric Ries’s Three Engines of Growth
Stickiness Virality Price
Approach
Math that
matters
Keep people
coming back.
Get customers
faster than you
lose them.
Make people invite
friends.
How many they
tell, how fast they
tell them.
Spend money to
get customers.
Customers are
worth more than
they cost.

Lean Analytics Stages
Empathy
Stickiness
Virality
Revenue
Scale
Stage
You’ve found a real, poorly-met need that a reachable market
faces.
You’ve ﬁgured out how to solve the problem in a way that users
will adopt, keep using, and pay for.
Your users and features fuel growth organically and artiﬁcially.
You’ve found a sustainable, scalable business with the right
margins in a healthy ecosystem.
Gate

Lean Analytics Stages
Empathy
Stickiness
Virality
Revenue
Scale
Stage
You’ve found a real, poorly-met need that a reachable market
faces.
You’ve ﬁgured out how to solve the problem in a way that users
will adopt, keep using, and pay for.
Your users and features fuel growth organically and artiﬁcially.
You’ve found a sustainable, scalable business with the right
margins in a healthy ecosystem.
Gate
Most projects/products/
startups fail here

Case Study: Jumping the gun on product
development
• Stage: Empathy/Stickiness
• Model: E-Commerce
• Originally tied to Instagram with
an “Insta-Order” feature

Case Study: Optimize for 1st purchase or
repeat orders?
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
With Insta-Order feature Without Insta-Order
• 2x transactions
• Lower bounce rate
• Sign-in goals increased

“THERE ARE NO SHORTCUTS TO
ANY PLACE WORTH GOING.”
- Beverly Sills

Case Study: Localmind hacks Twitter
(use a proxy to test your hypotheses)
BIGGEST RISK:
Would people be willing to answer
questions about a place in real-
time?

Case Study: Localmind hacks Twitter
(use a proxy to test your hypotheses)
Herbert Simon Tested results
• The response rate to tweeted questions was very high
• Good enough proxy to de-risk the solution and convince the team to continue
Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about the area
Would people answer questions?
• Before writing a line of code, they wanted to answer this question
• This was their biggest risk; if questions went unanswered, the experience would suck

Business models
• E-Commerce
• SaaS (freemium)
• Mobile app (gaming)
• Two-sided marketplace
• Media
• User-generated content

THE GOAL IS TO UNDERSTAND THE
CUSTOMER’S LIFECYCLE / JOURNEY
THROUGH EVERY TOUCH POINT
WITH YOUR PRODUCT.

Paid Direct WOM Search
Inherent
virality
Customer Acquisition Cost
VISITOR
User
FORMER USERS
Engaged user
Reactivate Trial over
Invite others
Paying customer
Disengaged
Account cancelled
Freemium / trial offer
Enrolment
Disengaged user
Cancel Cancel
Reactivate
FORMER CUSTOMERS
Billing info exp.
Resolution
Dissatisfied
Capacity Limit
Upselling
Signup conversion
rate
Free user disengagement
Freemium churn Reactivation
rate
User lifetime value Customer lifetime value
Trial abandonment
rate
DAU/WAU/MAU Paid conversion
Viral coefficient
Viral rate
Paid churn rate
Support data
Tiering
Upselling
rate
SaaS Customer
Lifecycle

Returning Paid Direct Search Viral
Customer Acquisition Cost
VISITOR
E-Commerce
Customer
Lifecycle
Navigation Search Reco Engine
1-time buyer
Cart
Additions
Conversion
Logistics, delays
Delivery
Enrolment
Call to Action
Sharing
Unsocial buyer
Sharing rate
Returning rate
Customer Lifetime Value
Open rate, engagement
Transaction size
Emphasis on maximizing cart
value, minimizing acquisition
costs
Bounced
Not interested
Abandoned
Bounce rate
Unsatisfied
Ratings, delivery
issues
Feature usage,
product discovery
Emphasis on
repurchase
rate, LTV

Case Study: WineExpress A/B tests what
really matters
Think of this over time…A B

Case Study: WineExpress A/B tests
what really matters
B
• 41% increase in revenue per
customer! (People bought a lot
more product.)
• Conversion also went up, but
was secondary in importance

All business models have issues
E-Commerce
SaaS
Mobile Apps
2-Sided Marketplace
Media
UCG
CAC vs. LTV — margins are usually small. A $10M e-commerce business is
often considered quite small.
Freemium requires tens of millions of free users. They can be expensive to
support. Will enough convert?
The average # of apps downloaded by North Americans per month is now 0.
Monetizing is incredibly hard. Popularity is ﬂeeting.
Chicken & egg problem. Supply and demand. How do you build up enough of
both?
Real monetization requires hundreds of millions of engaged visitors. People’s
attention is hard to capture and keep.
Content creation. Will it be good enough? Will enough people do it? Why?

YOU KNOW WHAT BUSINESS YOU’RE
IN AND WHAT STAGE YOU’RE AT.
NOW WHAT?

The One Metric That Matters
The business you’re in
E-Commerce SaaS Mobile 2-Sided Marketplace UCG Media
Thestageyou’reat
Empathy
Stickiness
Virality
Revenue
Scale
ONE METRIC THAT MATTERS

The One Metric That Matters
• The metric should indicate that your user experienced the
core value of the product
• It should reﬂect user’s engagement and activity level
• It should be the “one thing” that indicates the business is
heading in the right direction
• The metric ideally should be easy to understand &
communicate across teams
• The OMTM will change over time as the business evolves

Case study: Moz cuts down on metrics to
track
SaaS-based SEO toolkit in the Scale stage.
Focused on net adds.
Net adds up:
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds flat:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds down:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somewhere?
Is customer support falling apart?

Examples of the One Metric That Matters
# of transactions (for
merchants)
# of nights booked sales
total time reading
https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj
http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora
http://500.co/aircall-growth-uber/
monthly active users monthly recurring
revenue (MRR)

Metrics are like
squeeze toys

The Layer Cake of Metrics
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Department OMTM Department OMTM Department OMTM
OMTM: Business Help Indicator

Some interesting benchmarks
Growth
5% / week (revenue or
active users)
Time on site
17 minutes
Free to paid
2% of free users
Mobile ﬁle size
< 50MB
Engaged visitors
30% monthly users
10% daily users
Paid load time
< 5 seconds
Churn
2% / month
CLV:CAC
3:1

Case study: Solare draws a line in the sand

Case study: Solare discovers a leading
indicator
50 reservations by 5pm 250 covers that night
=

Draw a new line
Lean Analytics Cycle
Pivot or 
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data: 
ﬁnd a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a OMTM

Map your business model
1
2
Map your business as a systems diagram
Think about all the touch points of a user /
customer and how they interact with your
business (idea, product, people, etc.)
3
What are the riskiest areas / biggest areas of
uncertainty?
4
Where are you going to focus? Can you identify
the OMTM at each stage of your business?

THE END!
1
Data people need to be able to
“talk business”
So where are we at now?
• Understand the types of innovation and how a
company is approaching new product development
& business models
• Lean Startup applies beyond startups, but it’s not
easy
• Understand how the sausage is made (product
management)
• Product isn’t built in a vacuum; there’s a rigorous,
experiment-oriented process that helps
• Get involved in product roadmaps & feature
prioritization
• Data is both an input and a filter into the entire
process of building a business

THE END!
1
Data people need to be able to
“talk business”
So where are we at now?
• Understand the types of innovation and how a
company is approaching new product
development & business models
• Lean Startup applies beyond startups, but it’s not
easy
• Understand how the sausage is made (product
management)
• Product isn’t built in a vacuum; there’s a rigorous,
experiment-oriented process that helps
• Get involved in product roadmaps & feature
prioritization
• Data is both an input and a filter into the entire
process of building a business
2
Everyone else needs to be able to
“speak data”
• Data is the common language that everyone
needs to be reasonably fluent in
• We need to simplify the data (not what we track)
for others
• Data needs to be tied to real/core business goals
• The One Metric That Matters is a good construct
for forcing simplification and focus
• It’s important to understand the business you’re in
and the stage you’re at in order to identify the right
metrics to focus on

Once, a leader convinced others
in the absence of data.

Now, a leader knows
what questions to ask.

Alistair Croll
acroll@gmail.com
@acroll
Ben Yoskovitz
byosko@gmail.com
@byosko

Value Propositions
DESIGNING A BUSINESS MODEL

Lean Analytics - Communication Layer into the World of Business

More Related Content

What's hot (20)

Similar to Lean Analytics - Communication Layer into the World of Business (20)

Recently uploaded (20)

Lean Analytics - Communication Layer into the World of Business