Modern Product Data Workflows: How King Crushes New Product Development using Data-Driven Insights

© King.com Ltd 2018 – Commercially confidential
How King Crushes New Product Development
using Data-Driven Insights
Ian Thompson and
Zara Wells
Hannah Flynn
With:
Moderated by:
TO USE YOUR COMPUTER'S AUDIO:
When the webinar begins, you will be connected to audio
using your computer's microphone and speakers (VoIP). A
headset is recommended.
Webinar will begin:
11:00 am, PST
TO USE YOUR TELEPHONE:
If you prefer to use your phone, you must select "Use Telephone"
after joining the webinar and call in using the numbers below.
United States: +1 (562) 247-8422
Access Code: 864-244-421
Audio PIN: Shown after joining the webinar
--OR--

Looker is a complete data platform that offers data analytics and business
insights to every department, and easily integrates into applications to
deliver data directly into the decision-making process. Looker is powering
data-driven cultures at more than 1200 industry-leading and innovative
companies.

Click on the Questions panel to
interact with the presenters
www.productmanagementtoday.com/webinar-series/modern-product-data-workflows
www.projectmanagementupdate.com/webinar-series/modern-product-data-workflows
www.businessinnovationbrief.com/webinar-series/modern-product-data-workflows

About Ian Thompson
Ian Thompson is Head of Business Intelligence at King, where he focuses on all things Looker-related, data
modeling, analysis, and training Product Managers to take best advantage of their available tools. Prior to working
at King, he was a developer at Thomson Reuters where he was responsible for pharmaceuticals benchmarking
programs. He attended Bournemouth University.
About Zara Wells
Zara Wells is a Customer Success Manager at Looker, focusing on Strategic Accounts. She is a qualified Change
Management Practitioner. Prior to Looker, she was an account manager at Oracle for Bronto Software. She holds a BA
from Brunel University.

© King.com Ltd 2018 – Commercially confidential Page 6
About King

• We have developed more than 200 fun titles and our games can be played and
enjoyed all over the world.
• King had 270 million monthly active users for the quarter (Q2 2017)
• The company has been part of Activision Blizzard since February 2016
Page 7
We make great games
King has offices and studios
in Stockholm, London,
Barcelona, Bucharest,
Malmo, Berlin, Seattle, San
Francisco, Chicago, New
York, Los Angeles, Malta
and Tokyo.

The evolution of King
• Founded in 2003
• Originally games were
only available through our
site and portals including
AOL and Yahoo!
Online skill Social Mobile
• Launched first game on
Facebook in Q2 2011
• Launched first game on
mobile H2 2012

Some of our games

About Me

• Ian Thompson
• 5 years at King
• Head of Business Intelligence
Platform
• My focus
• Data & usage of data (how and
why)
• Researching & planning solutions
for problems which exist in the
business that can be solved by
data.
About Me

My Story
• BI (Business Intelligence)
Developer
• Qlikview - “Business
discovery platform”
2013 2015 2017
• ETL (extract transform and
load)/Data Modeling
Developer
• SQL, Bash & Jenkins
• BI Engineer & BI Platform
Product Owner
• Looker

• King analytics through the years
• 2012/13 Growth
• Org Structure 2013
• Case Studies
• AB Testing
• Level difficulty
• Soft Launches
• Org Structure 2017
• Looker @ King
• Case Studies Revisited
• What's next?
Page 13
Content

Growth
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014

2012/3

• Two
horizontal/organisational
data teams focusing on the
data warehouse and
reporting.
• Each with a single backlog
having to prioritise between
stakeholder work.
• The business talks to
warehousing team through
analysts/data scientists and
then through the reporting
team.
Page 16
2013 Org Structure
Data Warehouse Team
Reporting Team
Game
Team
Game
Performanc
e Unit
(Analyst/Dat
a Scientist
Finance
Marketing
Marketing
Analyst
CRM
CRM
Analysts/Da
ta Scientist

Case Studies

• Big Data & BI Tooling
• Push the cube into or through a BI tool
(QlikView/Tablaeu)
• Self service BI
Page 18
but first…OLAP Cubes

• Plan the test
• What are we testing
• What geography/segment
• What population size
• How to split the population
• How long do we run the test
for
• Hypothesis
Page 19
AB Testing
• What do we expect to
change with this test
• Appstore engagement
• Install volumes
• App starts on the device
• Push message clicks
• What do we NEED to
measure to (dis)prove our
Hypothesis
• Appstore visits
• Number of Installs
• Number of App starts
• Number of Push message
clicks*not a real AB Test
App Icon Badges

AB Testing
*not a real AB Test
Square Round Square_Party
Casenum 0 (control) 1 2
Population% 80.1 9.9 10
Key Value
Test Name Icon Badge
Test Version 1
Game CCS
Cases 3
No.
Assignments
200,000
Country UK, US, FR
Start Dt 2017-04-11
End Dt 2017-04-20
“Changing the app icon badge will
increase engagement and installs”
Store this
information
in the database
Test
Name
Test
Versi
on
Case
num
UserI
D
Icon
Badge
1 0 123
Icon
Badge
1 1 456
… … … …

AB Testing
Test Name Test
Version
Casenum Countr
y
Date num_da
u
num_appsta
rts
num_instal
ls
Icon Badge 1 0 UK 2018-08-01 772 1322 204
Icon Badge 1 0 US 2018-08-01 1465 2606 381
Icon Badge 1 0 FR 2018-08-01 420 6908 93
Icon Badge 1 1 UK 2018-08-02 61 113 23
Icon Badge 1 1 US 2018-08-02 157 298 59
Icon Badge 1 1 FR 2018-08-02 60 105 45
Icon Badge 1 2 UK 2018-08-03 82 99 21
Icon Badge 1 2 US 2018-08-03 111 284 21
Icon Badge 1 2 FR 2018-08-03 33 78 7
*not a real AB Test
AB Test – data cube:

AB Testing
1 2 3 4 5 6
Indexed DAU over time
Casenum0 Casenum1 Casenum2
*not a real AB Test

• What would be nice to look at now?
• Split by platform
• Spend
• Distribution of our assignment for
• Spend
• Engagement
Page 23
AB Testing • What do we expect to
change with this test
• Appstore engagement
• Install volumes
• App starts on the device
• Push message clicks
• What do we NEED to
measure to (dis)prove our
Hypothesis
• Appstore visits
• Number of Installs
• Number of App starts
• Number of Push message
clicks
*not a real AB Test
Data Team

Level Difficulty
• Hard
• Stuck on level - Churn
• Purchase “boosters”
• Convert to spender
• Increase spend
• Sense of
achievement/satisfaction when
won
• Virality
• Easy
• Keep player in our network
• Potentially convert to a payer in
the future
• Cross promote to other games
• Boring – no sense of
achievement/satisfaction

Level Difficulty

Level Difficulty
Level 65!
* as of 2012/3

Level Difficulty

Level Difficulty
* as of 2012/3

Level Difficulty
• Lets automate this and put it into a report
• All games
• All levels
• For dimensions and measures
• X,Y,Z…Z1,Z2…
Data Team
* as of 2012/3

• New games
• Test in smaller markets which behave like larger markets
• Eg. Ireland -> UK
• Eg. Canada -> US
• Up to 6 incremental playtest builds of the game
• Each focus on a major game feature or aspect of the game with retention being the
most important
• No monetisation features in the initial playtests
• Looking to improve specific metrics over the last playtest to achieve the greenlight
target
• Set time for each playtest to run and time between tests to make improvements
• Pre-determined number of installs allowed
• Categorise installs between organic, paid, cross promoted
Page 32
Playtest Games

Playtest Games
Data Scientist
If your new game in playtest was similar to Candy Crush Saga then you would benefit
from the reporting and data tools that had already been put in place. However any
features or aspects which were not similar would compete (and lose) for the data teams
attention.
* as of 2012/3

Playtest Games
* as of 2012/3
2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Retention
Distribution
Platform
% 2 day
Retention
14 day
% diff
% 7 day
Retention
14 day
% diff
% 30 day
Retention
14 day
% diff
Total 33.30% -0.70% 17.30% 0.00% 10.40% 3.50%
Google Play 43.50% -1.40% 22.70% -6.20% 14.80% -0.10%
iOS App
Store
45.70% 0.40% 26.20% -3.90% 15.80% 11.10%
Facebook 9.20% -16.70% 4.90% 24.20% 2.30% -0.20%
Standard KPI Reporting

2017

• Big Data
• Technology has improved so big data
is just….data
• Push the cube into or through a BI tool
(QlikView/Tablaeu)
• Read any data on the fly
• Self service BI
Page 36
OLAP Cubes

BI Platform
Game
Data Team
Finance & Marketing
Data Team
Game
Data Team
Intro
• Multiple vertical/stream data
teams focusing on the
warehouse and reporting
for a specific stakeholder.
• Each with a single backlog
having to prioritise single
stakeholder work.
• The business &
analysts/data scientists
have access to all data
teams and abilities but a
single point of contact.
Page 37
2017 Org Structure
Data Warehouse Abilities
Reporting Abilities
Game
Team &
Game
Performan
ce Unit
CRM
Marketing &
Marketing
Analysts/Da
ta Scientists
Finance &
Finance
Analysts
CRM &
Analysts/Da
ta Scientists
Core Warehouse Platform & Products

Looker @ King

BI Platform
Intro
Page 39
2017 Org Structure & Looker
• BI Platform - Small team (2 people)
• Focus on Looker adoption and understanding
• Data
• KPIs
• Tracking
• Dimensions
• Measures
• Training
• Workshops
• Tutorials/Documentation
• Face to face
• Tools & Performance
• Integration
• Debugging
• Generic data models

BI Platform
Intro
Page 40
2017 Org Structure & Looker
• The BI & data world has
changed, everyone who
wanted to, could interact &
alter what they did with the
data
• People felt empowered and
pushed on to produce
models and analysis
without their data teams
• Little to no path driven
analysis (data cubes), free
to explore whats, hows and
whys
The Business
Data Teams
• Data teams could really
focus on what's new
and important as minor
requests were
completed by the users
themselves
• Discussions shifted
from data problems to
business problems

Case Studies Revisited

• What would be nice to look at
now?
• Split by platform
• Spend
• Distribution of our assignment for
• Spend
• Engagement
Page 42
AB Testing
Data Team
Data Team/DS/End
User

• Data team builds foundations for an “AB Test Model” in
Looker
• Includes all the main data sets
• Simple and fast iterations
• Live prototyping during discussions
• End user contributions
• Add custom data sets specific to a game or AB Test
• Customise labels, descriptions, dashboards
• Analyse what is and is not used
Page 43
AB Testing

Level Difficulty

Playtest Games
Data Scientist
• Still ever so slightly true, but….
• DS spends more time “data sciencing”
• Discussions are more generic around solutions rather than solving a specific
problem
• Data problems are less common due to database speeds and the concept of
Looker
• Accessible for all not just DS
• Not reliant upon data teams or data skills

Playtest Games

What’s Next?

Sample pages
Page 53
What’s Next?
• Google Cloud Platform
• Take stock and tidyup!
• Centralise (only a little)
• Keep growing the eco-system

Data Driven Product Decisions
Zara Wells, Strategic Customer Success Manager

Monitoring Churn/ Retention

Impact of changes to the UI/ UX

Where to start
Plan Track Comms Wins

Q&A
Zara Wells
With: With:
Head of Business Intelligence, King
Linkedin page: /in/ian-thompson-93a5b216/
Twitter ID: @king_games
Website: king.com
Ian Thompson
Strategic Customer Success Manager, Looker
Linkedin page: /in/zara-wells-96659158/
Twitter ID: @LookerData
Website: looker.com
www.productmanagementtoday.com/webinar-series/modern-product-data-workflows
www.projectmanagementupdate.com/webinar-series/modern-product-data-workflows
www.businessinnovationbrief.com/webinar-series/modern-product-data-workflows

Modern Product Data Workflows: How King Crushes New Product Development using Data-Driven Insights

More Related Content

What's hot (20)

Similar to Modern Product Data Workflows: How King Crushes New Product Development using Data-Driven Insights (20)

More from Hannah Flynn (20)

Recently uploaded (20)

Modern Product Data Workflows: How King Crushes New Product Development using Data-Driven Insights